Overview¶
In this project, we will explore how to use Bayesian Networks (BN) and Hidden Markov Models (HMM) to predict price movements of front-month oil futures, with the end goal of demonstrating how this approach can provide a reliable signal for a profitable trading system.
In [1]:
from collections import Counter
import itertools
import json
import logging
import os
import requests
import time
import warnings
from dotenv import load_dotenv
from fredapi import Fred
import matplotlib.pyplot as plt
from myeia import API as meyeia_api
import networkx as nx
import numpy as np
import pandas as pd
from pgmpy.estimators import HillClimbSearch, BayesianEstimator, BIC, K2, BDeu
from pgmpy.factors.discrete import TabularCPD
from pgmpy.models import DiscreteBayesianNetwork
from pomegranate.distributions import Categorical
from pomegranate.hmm import DenseHMM
from scipy.stats import kurtosis, norm, skew
import seaborn as sns
from sklearn.metrics import (
confusion_matrix, ConfusionMatrixDisplay, f1_score,
precision_score, recall_score, accuracy_score,
)
import yfinance as yf
load_dotenv() # Load our secrets from the .env file
# We'll be fitting a lot of pgmpy models, so we will suppress those annoying logs
logging.getLogger("pgmpy").setLevel(logging.CRITICAL + 1)
# pgmpy raises a lot of these from pandas, so we will suppress the pandas FutureWarning
warnings.filterwarnings("ignore", category=FutureWarning)
BARS_PER_YEAR = 251 # Average trading days per year, rounded to nearest integer
START_DATE = "1900-01-01"
END_DATE = None
LAST_ANALYSIS_DATE = "2025-09-01"
FRED_SESSION = Fred(api_key=os.getenv("FRED_API_KEY"))
EIA_SESSION = meyeia_api(token=os.getenv("EIA_API_KEY"))
TODAY = pd.to_datetime('today')
THIS_MONTH = pd.to_datetime(TODAY.strftime('%Y-%m'))
TARGET_FEATURE = 'CL=F' # WTI Oil front month futures price
TARGET_LABEL = 'target' # name we will use for the target feature
TARGET_HORIZON_DAYS = 5 # predict return over next 5 TRADING days
def get_freq_from_index(df: pd.DataFrame) -> pd.Timestamp:
"""Using the data's datetime index, classify its sampling frequency as
'annually', 'monthly', 'weekly', 'daily', or 'intraday'.
"""
x = np.nanmean(df.index.diff().days)
if 245 < x:
freq = 'annually'
elif 28 <= x <= 31:
freq = 'monthly'
elif 3 <= x <= 7: # Some trading weeks are short due to holidays
freq = 'weekly'
elif 1 <= x < 3: # Weekends might shift the average within [1, 3)
freq = 'daily'
else:
raise ValueError("Only daily or higher timeframes are permitted.")
return freq
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
Performance Metrics¶
Before we begin building and evaluating the model, we should determine how we want to measure its performance. Ultimately, our goal is to predict whether WTI Oil futures will increase or decrease in price over the next 5 trading days. Since there are only two possible outcomes, this is a binary classification problem. We will want to measure:
- Accuracy as our primary metric, because a long-and-short strategy will depend heavily on our overall rate of correct guesses.
- Precision, because a long-only strategy will depend heavily on how many of our "increase" guesses are correct.
- Recall and the F1-score, because an analyst using a long-only strategy might be interested in how many opportunities they missed.
All of these metrics are implemented in SciPy, so we'll just import those functions.
Additionally, for any set of data that we try to predict, we want to know the answer to this question:
- What are the chances our model achieved its measured accuracy purely by luck, rather than by identifying a legitimate pattern in the market?
To answer this question, we can use a permutation test where we simulate a bunch of random scenarios with the same proportions of answers as our data, but shuffled around, and then compare our model's accuracy in the real world against its accuracy in the simulated random worlds.
Finally, we will want to compare a couple of toy trading strategies against a baseline buy-and-hold strategy, so it will be helpful to compute:
- Final portfolio return.
- Maximum percent drawdown as a measure of value at risk.
- Calmar Ratio as a comparison of reward and risk.
- Omega Ratio as a probability-weighted average of gains versus losses. We will use a threshold of $\theta = 0$.
- Sharpe Ratio, since it is widely used (despite its drawbacks).
- Probablistic Sharpe Ratio, which tells us the probability that a strategy's true Sharpe Ratio outperforms a benchmark Sharpe Ratio. We will use the buy-and-hold baseline strategy to compute the benchmark Sharpe. We're looking for PSR > 0.95 to dismiss the null hypothesis of not exceeding the benchmark.
- Deflated Sharpe Ratio, which tells us the probability that a strategy's true Sharpe Ratio outperforms the expected maximum Sharpe Ratio caused by multiple testing. The more strategies we test, the easier it is to find a random strategy with a high Sharpe Ratio; this metric helps us avoid that issue. We're looking for DSR > 0.95 to dismiss the null hypothesis of not exceeding the random noise Sharpe Ratio.
In [2]:
def permutation_test(
y_pred: np.ndarray,
y_true: np.ndarray,
n_iter: int=10000,
seed: int | None=None
) -> np.float64:
"""Conducts a permutation test under the null hypothesis that no structural information
in y_true was modeled in y_pred. The test shuffles y_true for a total of n_iter different
combinations (but does not shuffle y_pred) and computes the accuracy of y_pred with each
shuffled version of y_true. Returns the distribution of simulated accuracy metrics and the
model's percentile within that distribution.
"""
rng = np.random.default_rng(seed=None)
x = np.tile(y_true.flatten(), (n_iter, 1)).T # (n_answers, n_iter)
rng.permuted(x, axis=0, out=x)
for elem in np.unique(x):
mu = (x == elem).mean(axis=0)
assert np.all(mu == mu[0]), f"Distribution of {elem} not equal across all columns."
accuracies = (x == y_pred.reshape(-1, 1)).mean(axis=0)
model_accuracy = accuracy_score(y_true=y_true, y_pred=y_pred)
percentile = max(
(accuracies >= model_accuracy).mean(),
1 / n_iter # If none of the simulations beat the model, the minimum p-value is 1/n_iter.
)
return percentile, accuracies
def compute_final_portfolio_pnl(portfolio: np.ndarray) -> np.float64:
"""Computes the final portfolio profit or loss as a fraction of initial
portfolio value.
"""
return portfolio[-1] / portfolio[0] - 1
def compute_max_drawdown(portfolio: np.ndarray) -> np.float64:
"""Computes the maximum drawdown from a sequence of portfolio values."""
cumulative_max = np.maximum.accumulate(portfolio)
drawdowns = (portfolio - cumulative_max) / cumulative_max
max_drawdown = drawdowns.min()
return max_drawdown
def compute_calmar_ratio(
portfolio: np.ndarray,
rf_annualized: float | np.ndarray=0.,
bars_per_year: int=BARS_PER_YEAR
) -> np.float64:
"""Compute the Calmar Ratio. Note that rf_annualized does not need
to be the same length as portfolio.
"""
n_years = portfolio.flatten().shape[0] / bars_per_year
cagr = (portfolio[-1] / portfolio[0]) ** (1 / n_years) - 1
cagr_excess = cagr - rf_annualized.mean()
max_drawdown = compute_max_drawdown(portfolio)
if max_drawdown == 0:
return np.inf
return cagr_excess / np.abs(max_drawdown)
def compute_omega_ratio(
portfolio: np.ndarray,
threshold_annualized: float | np.ndarray=0.,
bars_per_year: int=BARS_PER_YEAR
) -> np.float64:
"""Computes the Omega Ratio. If threshold_annualized is an array, it must have the same length
as portfolio.
"""
if isinstance(threshold_annualized, np.ndarray):
assert portfolio.shape[0] == threshold_annualized.shape[0]
threshold_annualized = threshold_annualized[1:]
threshold = (1 + threshold_annualized) ** (1 / bars_per_year) - 1
r = np.diff(portfolio) / portfolio[:-1]
gains = (r - threshold)[r > threshold]
losses = (threshold - r)[r < threshold]
if losses.sum() == 0:
return np.inf
return gains.sum() / losses.sum()
def compute_sharpe_ratio(
portfolio: np.ndarray,
rf_annualized: float | np.ndarray=0.,
bars_per_year: int=BARS_PER_YEAR,
scale_period: int=BARS_PER_YEAR
) -> np.float64:
"""Compute the Sharpe Ratio, scaled by scale_period. If rf_annualized is an array, it
must have the same length as portfolio.
"""
if isinstance(rf_annualized, np.ndarray):
assert portfolio.shape[0] == rf_annualized.shape[0]
rf_annualized = rf_annualized[1:]
rf = (1 + rf_annualized) ** (1 / bars_per_year) - 1
r = np.diff(portfolio) / portfolio[:-1]
r_excess = r - rf
sr = r_excess.mean() / r_excess.std()
sr_scaled = sr * np.sqrt(scale_period)
return sr_scaled
def compute_psr(
portfolio: np.ndarray,
benchmark_portfolio: np.ndarray,
rf_annualized: float | np.ndarray=0.,
bars_per_year: int=BARS_PER_YEAR,
scale_period: int=BARS_PER_YEAR,
) -> np.float64:
"""Compute the Probabilistic Sharpe Ratio. Assumes that portfolio and benchmark_portfolio
have the same length and sampling frequency. If rf_annualized is an array, it must also
have the same length.
"""
assert portfolio.shape[0] == benchmark_portfolio.shape[0]
if isinstance(rf_annualized, np.ndarray):
assert portfolio.shape[0] == rf_annualized.shape[0]
rf = (1 + rf_annualized) ** (1 / bars_per_year) - 1
r_excess = np.diff(portfolio) / portfolio[:-1] - rf[1:]
sr_portfolio = compute_sharpe_ratio(
portfolio=portfolio,
rf_annualized=rf,
bars_per_year=bars_per_year,
scale_period=scale_period
)
sr_benchmark = compute_sharpe_ratio(
portfolio=benchmark_portfolio,
rf_annualized=rf,
bars_per_year=bars_per_year,
scale_period=scale_period
)
gamma3 = skew(r_excess, bias=False)
gamma4 = kurtosis(r_excess, bias=False, fisher=True)
t = len(r_excess)
numer = (sr_portfolio - sr_benchmark) * np.sqrt(t - 1)
radicand = 1 - gamma3 * sr_portfolio + ((gamma4 - 1) / 4) * sr_portfolio**2
if radicand < 0:
denom = 1e-9
else:
denom = np.sqrt(radicand)
return norm.cdf(numer / denom)
def compute_dsr(
portfolio: np.ndarray,
n_tests: int,
rf_annualized: float | np.ndarray=0.,
bars_per_year: int=BARS_PER_YEAR,
scale_period: int=BARS_PER_YEAR,
) -> np.float64:
"""Compute the Deflated Sharpe Ratio. If rf_annualized is an array, it must have
the same length as portfolio. DSR differs from PSR in that, while PSR computes the
probability of the portfolio's true SR exceeding that of the benchmark, DSR computes
the probability of the portfolio's true SR exceeding that of random noise introduced
by testing multiple strategies. Should be computed for each strategy tested.
"""
if isinstance(rf_annualized, np.ndarray):
assert portfolio.shape[0] == rf_annualized.shape[0]
rf = (1 + rf_annualized) ** (1 / bars_per_year) - 1
r_excess = np.diff(portfolio) / portfolio[:-1] - rf[1:]
sr_portfolio = compute_sharpe_ratio(
portfolio=portfolio,
rf_annualized=rf,
bars_per_year=bars_per_year,
scale_period=scale_period
)
t = len(r_excess)
sr_max = np.sqrt(np.log(n_tests) / (t - 1))
gamma3 = skew(r_excess, bias=False)
gamma4 = kurtosis(r_excess, bias=False, fisher=True)
numer = (sr_portfolio - sr_max) * np.sqrt(t - 1)
radicand = 1 - gamma3 * sr_portfolio + ((gamma4 - 1) / 4) * sr_portfolio**2
if radicand < 0:
denom = 1e-9
else:
denom = np.sqrt(radicand)
return norm.cdf(numer / denom)
def get_portfolio_metrics(
portfolio: np.ndarray,
benchmark_portfolio: np.ndarray,
n_tests: int,
rf_annualized: float | np.ndarray=0.,
bars_per_year: int=BARS_PER_YEAR
) -> dict:
"""Computes all portfolio metrics, returning them in a dictionary."""
final_pnl = compute_final_portfolio_pnl(portfolio=portfolio)
md = compute_max_drawdown(portfolio=portfolio)
calmar = compute_calmar_ratio(
portfolio=portfolio,
rf_annualized=rf_annualized,
bars_per_year=BARS_PER_YEAR
)
omega = compute_omega_ratio(
portfolio=portfolio,
threshold_annualized=rf_annualized,
bars_per_year=BARS_PER_YEAR
)
sharpe = compute_sharpe_ratio(
portfolio=portfolio,
rf_annualized=rf_annualized,
bars_per_year=BARS_PER_YEAR,
scale_period=BARS_PER_YEAR
)
psr = compute_psr(
portfolio=portfolio,
benchmark_portfolio=benchmark_portfolio,
rf_annualized=rf_annualized,
bars_per_year=BARS_PER_YEAR,
scale_period=BARS_PER_YEAR,
)
dsr = compute_dsr(
portfolio=portfolio,
n_tests=n_tests,
rf_annualized=rf_annualized,
bars_per_year=BARS_PER_YEAR,
scale_period=BARS_PER_YEAR,
)
result = {
'final_pnl': final_pnl,
'max_drawdown': md,
'calmar': calmar,
'omega': omega,
'sharpe': sharpe,
'psr': psr,
'dsr': dsr
}
return result
ETL¶
Define Features¶
In [3]:
yf_features = [
'^SPX', # S&P 500 index
'BZ=F', # Brent Crude Oil front month futures price
'CL=F', # WTI Oil front month futures price
'CVX', # Chevron stock price
'SHEL', # Shell PLC stock price
'XLE', # Energy sector of S&P 500
'XOM', # Exxon Mobile stock price
]
fred_features = [
'A24ATI', # Manufacturers' Total Inventories: Petroleum Refineries
'CAPG211S', # Industrial Capacity: Mining: Oil and Gas Extraction
'CAPUTLG211S', # Capacity Utilization: Mining: Oil and Gas Extraction
'CPIENGSL', # Consumer Price Index for All Urban Consumers: Energy in U.S. City Average
'IPG21112S', # Industrial Production: Mining: Crude Oil
'IPN213111N', # Industrial Production: Mining: Drilling Oil and Gas Wells
'INDPRO', # Industrial Production: Total Index
'OVXCLS', # CBOE Crude Oil ETF Volatility Index
'PCU211211', # Producer Price Index by Industry: Oil and Gas Extraction
'WTISPLC', # Spot Crude Oil Price: West Texas Intermediate
]
eia_features = [
# Total OPEC Petroleum Supply (million barrels per day)
dict(
route="steo",
series="PAPR_OPEC",
frequency="monthly",
facet="seriesId",
),
# U.S. Crude Oil Proved Reserves (Million Barrels) - up to 2023
dict(
route="petroleum/crd/pres",
series="RCRR01NUS_1",
frequency="annual",
facet="series",
),
# U.S. Federal Offshore Crude Oil Proved Reserves (Million Barrels) - up to 2023
dict(
route="petroleum/crd/pres",
series="RCRR01R48F_1",
frequency="annual",
facet="series",
),
# U.S. Product Supplied of Finished Motor Gasoline (Thousand Barrels per Day)
dict(
route="petroleum/cons/wpsup",
series="WGFUPUS2",
frequency="weekly",
facet="series",
),
]
Extract Data from Online Sources¶
Since Yahoo Finance, FRED, and EIA have various sorting schemes, we will handle datetime index sorting immediately and all other data transformations later.
In [4]:
feature_freq_lag = []
print("Downloading Yahoo Finance data...")
yf_df = yf.download(
yf_features, start=START_DATE, end=END_DATE, auto_adjust=True
).sort_values(by='Date', ascending=True)['Close']
print("Downloading FRED data...")
fred_datalist = []
for feature in fred_features:
tmp = pd.DataFrame(FRED_SESSION.get_series(feature).sort_index(ascending=True))
tmp.columns = [feature]
tmp.index = pd.to_datetime(tmp.index)
fred_datalist.append(tmp)
ffl = {
'feature': feature,
'frequency': get_freq_from_index(tmp),
'lag [days]': (TODAY - tmp.index[-1]).days
}
feature_freq_lag.append(ffl)
print("Downloading EIA data...")
eia_datalist = []
for feature_dict in eia_features:
tmp = EIA_SESSION.get_series_via_route(**feature_dict).sort_values(by='Date', ascending=True)
tmp.columns = [feature_dict['series']]
eia_datalist.append(tmp)
ffl = {
'feature': feature_dict['series'],
'frequency': get_freq_from_index(tmp),
'lag [days]': (TODAY - tmp.index[-1]).days
}
feature_freq_lag.append(ffl)
datalist = [yf_df] + fred_datalist + eia_datalist
ffl_df = pd.DataFrame(feature_freq_lag)
display(ffl_df)
Downloading Yahoo Finance data...
[*********************100%***********************] 7 of 7 completed
Downloading FRED data... Downloading EIA data...
| feature | frequency | lag [days] | |
|---|---|---|---|
| 0 | A24ATI | monthly | 65 |
| 1 | CAPG211S | monthly | 65 |
| 2 | CAPUTLG211S | monthly | 65 |
| 3 | CPIENGSL | monthly | 65 |
| 4 | IPG21112S | monthly | 65 |
| 5 | IPN213111N | monthly | 65 |
| 6 | INDPRO | monthly | 65 |
| 7 | OVXCLS | daily | 1 |
| 8 | PCU211211 | monthly | 65 |
| 9 | WTISPLC | monthly | 34 |
| 10 | PAPR_OPEC | monthly | -453 |
| 11 | RCRR01NUS_1 | annually | 1707 |
| 12 | RCRR01R48F_1 | annually | 1707 |
| 13 | WGFUPUS2 | weekly | 6 |
The table above shows that most of the economic data is sampled on a monthly basis, but two are sampled annually, one weekly, and one daily. These series will have to be shifted by different amounts to account for their different sampling frequencies and lags. Surprisingly, one of the feature (PAPR_OPEC) has a negative lag, indicating that it's most recently available measurement corresponds to a point in the future. A closer look at the EIA's description of this feature explains why: it is a forecast of OPEC's petroleum supply provided by the EIA's Short Term Energy Outlook. Since none of our other features contain information corresponding with future timestamps, we will limit the availability of this one feature up to the present day within our model -- meaning that our model will only be able to access up to the prediction that was previously made for the present day. That said, it might be interesting to explore the impact of the one-month PAPR_OPEC forecast on our model's predictions in a later iteration of this analysis.
View Data to Check for Issues¶
Ideally, we would want to sequester our test set before viewing the remaining data, but it's more important for us to verify that there are no large gaps or other issues with data continuity. Since we will be cleaning and transforming our data anyway, a quick sanity-checking glance at the raw data poses little risk of influencing our models.
In [5]:
for d in datalist:
d.plot()
plt.yscale('log')
plt.show()
Clean/Transform data¶
In [6]:
def create_target_feature(
df: pd.DataFrame,
target_feature: str,
target_label: str,
target_horizon_days: int
) -> pd.DataFrame:
"""Creates the target feature from the provided dataframe column."""
res = pd.DataFrame()
res[target_label] = df[target_feature].pct_change(periods=target_horizon_days, fill_method=None)
return res
def handle_lagged_data_availability(
datasets: pd.DataFrame,
ffl_df: pd.DataFrame=ffl_df,
target_label: str=TARGET_LABEL
) -> pd.DataFrame:
"""To account for EIA and FRED data becoming available weeks or months after the periods they
represent, we will offset these time series with a lag that estimates the delay between
occurence and measurement. We will use the feature-frequency-lag DataFrame (ffl_df) to do this.
For the Yahoo Finance data, we will assume that trades take place during trading hours, meaning
that closing prices are available up to and including the previous day. Accordingly, we will shift
all Yahoo Finance data forward by one day to reflect this one-day lag.
"""
freq_dict = {
'annually': {'n_days': 365, 'kwarg': 'YS'},
'monthly': {'n_days': 30, 'kwarg': 'MS'},
'weekly': {'n_days': 7, 'kwarg': 'W'},
'daily': {'n_days': 1, 'kwarg': 'D'},
}
new_datalist = []
for i in range(len(datasets)):
for feature in datasets[i].columns:
if feature in ffl_df['feature'].values:
ffl_idx = ffl_df[ffl_df['feature']==feature].index[0]
ffl_freq = ffl_df.loc[ffl_idx, 'frequency']
freq = freq_dict[ffl_freq]['kwarg']
lag = ffl_df.loc[ffl_idx, 'lag [days]']
if lag < 0: # Forward-looking or current data becomes usable in the period for which it is posted.
n_shift = 0
else: # Backward-looking data may lag by multiple periods.
n_days = freq_dict[ffl_freq]['n_days']
n_shift = (lag // n_days) + 1
else:
n_shift = 1
freq = 'D'
shifted_feature = datasets[i][feature].shift(periods=n_shift, freq=freq)
new_datalist.append(shifted_feature)
return new_datalist
def merge_pct_changes_to_single_timeline(
main_df: pd.DataFrame,
datasets: list,
) -> pd.DataFrame:
"""Computes the percent change for all DataFrames in datasets, forward fills onto daily periods,
then merges onto the timeline provided by main_df.
"""
for df in datasets:
pc = df.pct_change(periods=1, fill_method=None).resample('D').ffill()
main_df = main_df.join(pc, how='left')
return main_df
def remove_duplicate_indices(df: pd.DataFrame) -> pd.DataFrame:
"""If there are duplicated timestamp indices, remove the duplicates and keep only the first."""
return df[~df.index.duplicated(keep='first')]
def handle_missing_values(df: pd.DataFrame) -> pd.DataFrame:
"""Forward fill missing values. This reflects the fact that, in a live trading environment,
we would have acesss to the most recently available data.
"""
return df.ffill().dropna()
def handle_predicted_values(df: pd.DataFrame) -> pd.DataFrame:
"""Since the EIA Short Term Energy Outlook data includes forecasts, we must limit our data
to timestamps up to the present day.
"""
return df.loc[:LAST_ANALYSIS_DATE]
clean_df = create_target_feature(
df=yf_df,
target_feature=TARGET_FEATURE,
target_label=TARGET_LABEL,
target_horizon_days=TARGET_HORIZON_DAYS
)
lagged_datalist = handle_lagged_data_availability(
datasets=datalist,
ffl_df=ffl_df,
target_label=TARGET_LABEL
)
clean_df = merge_pct_changes_to_single_timeline(
main_df=clean_df,
datasets=lagged_datalist,
)
preprocessing_pipeline = [
remove_duplicate_indices,
handle_missing_values,
handle_predicted_values,
]
for func in preprocessing_pipeline:
clean_df = func(clean_df)
clean_df
Out[6]:
| target | BZ=F | CL=F | CVX | SHEL | XLE | XOM | ^SPX | A24ATI | CAPG211S | ... | IPG21112S | IPN213111N | INDPRO | OVXCLS | PCU211211 | WTISPLC | PAPR_OPEC | RCRR01NUS_1 | RCRR01R48F_1 | WGFUPUS2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Date | |||||||||||||||||||||
| 2007-08-01 | 0.008566 | 0.017296 | 0.017962 | -0.013537 | -0.004490 | -0.009759 | -0.010231 | -0.012647 | 0.033729 | 0.000431 | ... | -0.000059 | 0.000836 | 0.000538 | 0.106789 | 0.043174 | 0.063347 | -0.000850 | 0.010291 | 0.035988 | -0.002684 |
| 2007-08-02 | 0.025484 | -0.022064 | -0.021481 | -0.001055 | 0.006764 | -0.002608 | 0.008340 | 0.007243 | 0.033729 | 0.000431 | ... | -0.000059 | 0.000836 | 0.000538 | 0.017573 | 0.043174 | 0.063347 | -0.000850 | 0.010291 | 0.035988 | -0.002684 |
| 2007-08-03 | -0.019995 | 0.005441 | 0.004312 | -0.015029 | -0.011112 | -0.006830 | -0.007689 | 0.004359 | 0.033729 | 0.000431 | ... | -0.000059 | 0.000836 | 0.000538 | -0.031281 | 0.043174 | 0.063347 | -0.000850 | 0.010291 | 0.035988 | -0.002684 |
| 2007-08-06 | -0.062085 | -0.013332 | -0.017955 | -0.034211 | -0.027571 | -0.032626 | -0.036394 | -0.026586 | 0.033729 | 0.000431 | ... | -0.000059 | 0.000836 | 0.000538 | -0.012129 | 0.043174 | 0.063347 | -0.000850 | 0.010291 | 0.035988 | -0.009004 |
| 2007-08-07 | -0.074031 | -0.047893 | -0.045310 | 0.012343 | 0.020290 | 0.005747 | 0.017787 | 0.024151 | 0.033729 | 0.000431 | ... | -0.000059 | 0.000836 | 0.000538 | -0.012129 | 0.043174 | 0.063347 | -0.000850 | 0.010291 | 0.035988 | -0.009004 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 2025-08-25 | 0.021760 | 0.000887 | 0.002204 | 0.016908 | 0.011450 | 0.020375 | 0.018768 | 0.015186 | -0.016920 | 0.003026 | ... | 0.004047 | -0.026024 | 0.000760 | -0.037730 | -0.074273 | 0.096510 | 0.006747 | -0.189088 | -0.135802 | 0.045012 |
| 2025-08-26 | 0.014435 | 0.015798 | 0.017908 | 0.000126 | 0.006819 | 0.002610 | 0.004134 | -0.004266 | -0.016920 | 0.003026 | ... | 0.004047 | -0.026024 | 0.000760 | -0.037730 | -0.074273 | 0.096510 | 0.006747 | -0.189088 | -0.135802 | 0.045012 |
| 2025-08-27 | 0.014871 | -0.022965 | -0.023920 | -0.005310 | -0.002167 | -0.001584 | -0.002237 | 0.004134 | -0.016920 | 0.003026 | ... | 0.004047 | -0.026024 | 0.000760 | 0.029325 | -0.074273 | 0.096510 | 0.006747 | -0.189088 | -0.135802 | 0.045012 |
| 2025-08-28 | 0.017002 | 0.012348 | 0.014229 | 0.011947 | 0.000679 | 0.011221 | 0.011301 | 0.002391 | -0.016920 | 0.003026 | ... | 0.004047 | -0.026024 | 0.000760 | 0.079706 | -0.074273 | 0.096510 | 0.006747 | -0.189088 | -0.135802 | 0.045012 |
| 2025-08-29 | 0.005498 | 0.008376 | 0.007015 | 0.000502 | 0.002985 | 0.007622 | 0.005321 | 0.003157 | -0.016920 | 0.003026 | ... | 0.004047 | -0.026024 | 0.000760 | -0.053068 | -0.074273 | 0.096510 | 0.006747 | -0.189088 | -0.135802 | 0.045012 |
4553 rows × 22 columns
If today is Fri 2025-08-15, then 'CL=F' represents the percent change at close of day yesterday (Wed 08-13 to Thurs 08-14) and 'target' represents the percent change over target_horizon_days starting from close of day yesterday (Thurs 08-14 to Thurs 08-21, assuming target_horizon_days = 5 trading days).
Train/Val/Test Split¶
Since our data are all ordered by time, we will take contiguous segments as our training, validation, and test sets.
- Using our training set, we will derive estimates for our population distributions.
- Using our validation set, we will iteratively select well-performing models and try to improve their performance. Information from the validation set will leak into our model with each iteration/comparison, so we will try to limit the number of times we retrain our model after exposing it to the validation set.
- The test set will be used exactly once on the final model. We will simulate a paper trading environment with walk-forward testing. The output will be either a "pass" or "fail" indication for our final model.
In [7]:
val_start_date = '2019-01-01'
test_start_date = '2023-01-01'
train_df = clean_df.loc[:val_start_date]
val_df = clean_df.loc[val_start_date:test_start_date]
test_df = clean_df.loc[test_start_date:]
print(f"Train Set: {train_df.index[0].date()} to {train_df.index[-1].date()} = {train_df.shape[0]} pts")
print(f"Val Set: {val_df.index[0].date()} to {val_df.index[-1].date()} = {val_df.shape[0]} pts")
print(f"Test Set: {test_df.index[0].date()} to {test_df.index[-1].date()} = {test_df.shape[0]} pts")
Train Set: 2007-08-01 to 2018-12-31 = 2875 pts Val Set: 2019-01-02 to 2022-12-30 = 1008 pts Test Set: 2023-01-03 to 2025-08-29 = 670 pts
EDA on Entire Training Set¶
In [8]:
train_df.describe()
Out[8]:
| target | BZ=F | CL=F | CVX | SHEL | XLE | XOM | ^SPX | A24ATI | CAPG211S | ... | IPG21112S | IPN213111N | INDPRO | OVXCLS | PCU211211 | WTISPLC | PAPR_OPEC | RCRR01NUS_1 | RCRR01R48F_1 | WGFUPUS2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | ... | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 | 2875.000000 |
| mean | 0.000396 | -0.000053 | 0.000095 | 0.000367 | 0.000280 | 0.000185 | 0.000150 | 0.000262 | 0.001922 | 0.004927 | ... | 0.006363 | 0.000324 | 0.000186 | -0.002020 | 0.000925 | 0.004866 | 0.000697 | 0.037759 | 0.005425 | 0.000333 |
| std | 0.051485 | 0.021691 | 0.024231 | 0.016979 | 0.017736 | 0.018336 | 0.015307 | 0.012679 | 0.047798 | 0.003546 | ... | 0.029207 | 0.045522 | 0.007474 | 0.047152 | 0.076196 | 0.088078 | 0.010368 | 0.079047 | 0.061344 | 0.030860 |
| min | -0.268150 | -0.103678 | -0.122478 | -0.124891 | -0.109153 | -0.144438 | -0.139526 | -0.090350 | -0.242779 | -0.001102 | ... | -0.211945 | -0.161877 | -0.043858 | -0.355903 | -0.261930 | -0.285864 | -0.040988 | -0.103016 | -0.086326 | -0.131559 |
| 25% | -0.028070 | -0.009928 | -0.012070 | -0.007334 | -0.007887 | -0.007668 | -0.006746 | -0.004134 | -0.023938 | 0.002151 | ... | -0.003349 | -0.006101 | -0.003167 | -0.027891 | -0.042094 | -0.041138 | -0.004157 | -0.034661 | -0.046631 | -0.015903 |
| 50% | 0.001458 | 0.000000 | 0.000358 | 0.000783 | 0.000728 | 0.000419 | 0.000106 | 0.000549 | 0.004887 | 0.004111 | ... | 0.006516 | 0.009096 | 0.000948 | -0.004775 | 0.005804 | 0.012599 | 0.001246 | 0.018062 | 0.001169 | 0.000771 |
| 75% | 0.028569 | 0.009917 | 0.011947 | 0.008396 | 0.008714 | 0.008735 | 0.007147 | 0.005573 | 0.033327 | 0.008631 | ... | 0.015732 | 0.028496 | 0.004087 | 0.020663 | 0.048988 | 0.060882 | 0.006629 | 0.124988 | 0.057904 | 0.017764 |
| max | 0.312712 | 0.135492 | 0.178329 | 0.208542 | 0.170246 | 0.164747 | 0.171905 | 0.115800 | 0.098569 | 0.011352 | ... | 0.185923 | 0.068748 | 0.013715 | 0.387892 | 0.198767 | 0.238456 | 0.030648 | 0.150128 | 0.106762 | 0.100602 |
8 rows × 22 columns
In [9]:
train_df.cov()
Out[9]:
| target | BZ=F | CL=F | CVX | SHEL | XLE | XOM | ^SPX | A24ATI | CAPG211S | ... | IPG21112S | IPN213111N | INDPRO | OVXCLS | PCU211211 | WTISPLC | PAPR_OPEC | RCRR01NUS_1 | RCRR01R48F_1 | WGFUPUS2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| target | 0.002651 | 4.179646e-04 | 5.166765e-04 | 1.948622e-04 | 2.168907e-04 | 2.495577e-04 | 1.484036e-04 | 1.065150e-04 | -1.362996e-04 | -1.210563e-05 | ... | 0.000097 | -2.029062e-04 | 2.100298e-05 | -0.000304 | -0.000155 | 2.631618e-04 | 2.449549e-05 | -6.014130e-05 | -0.000180 | -1.656708e-05 |
| BZ=F | 0.000418 | 4.704827e-04 | 4.463855e-04 | 1.806232e-04 | 1.903207e-04 | 2.209369e-04 | 1.420994e-04 | 9.248745e-05 | 1.815885e-07 | -1.663361e-06 | ... | 0.000029 | -3.309475e-05 | 8.117655e-06 | -0.000043 | -0.000014 | 8.061695e-05 | 5.934010e-06 | 5.148005e-06 | -0.000033 | 1.519193e-06 |
| CL=F | 0.000517 | 4.463855e-04 | 5.871483e-04 | 2.057230e-04 | 2.159511e-04 | 2.513348e-04 | 1.646715e-04 | 1.028059e-04 | -1.970058e-05 | -2.318434e-06 | ... | 0.000022 | -4.207374e-05 | 4.476718e-06 | -0.000056 | -0.000038 | 4.253496e-05 | 5.893857e-06 | -1.297691e-05 | -0.000037 | 3.804534e-07 |
| CVX | 0.000195 | 1.806232e-04 | 2.057230e-04 | 2.882765e-04 | 2.383802e-04 | 2.798419e-04 | 2.243228e-04 | 1.680362e-04 | -8.950196e-06 | -8.677502e-07 | ... | -0.000005 | 5.558130e-06 | 2.325853e-06 | -0.000009 | -0.000017 | -2.175281e-06 | -1.338520e-06 | -9.445202e-06 | -0.000008 | -1.877971e-06 |
| SHEL | 0.000217 | 1.903207e-04 | 2.159511e-04 | 2.383802e-04 | 3.145531e-04 | 2.652241e-04 | 2.055943e-04 | 1.631752e-04 | -6.468652e-06 | -5.672107e-07 | ... | -0.000011 | -5.671556e-06 | 3.420240e-07 | -0.000012 | -0.000002 | 2.613584e-07 | 9.041412e-07 | 1.618715e-06 | -0.000012 | -6.157678e-06 |
| XLE | 0.000250 | 2.209369e-04 | 2.513348e-04 | 2.798419e-04 | 2.652241e-04 | 3.362266e-04 | 2.433714e-04 | 1.925885e-04 | -5.006746e-06 | -6.617963e-07 | ... | -0.000004 | -5.182760e-06 | 2.556412e-06 | -0.000019 | -0.000009 | 1.443986e-05 | -3.348231e-07 | -1.665236e-05 | -0.000014 | -7.137386e-06 |
| XOM | 0.000148 | 1.420994e-04 | 1.646715e-04 | 2.243228e-04 | 2.055943e-04 | 2.433714e-04 | 2.343121e-04 | 1.505682e-04 | -4.471400e-06 | -3.811469e-07 | ... | -0.000006 | 8.773246e-07 | 1.206808e-06 | -0.000011 | -0.000016 | -7.584528e-06 | 7.432304e-07 | -1.546609e-05 | -0.000003 | -8.471577e-06 |
| ^SPX | 0.000107 | 9.248745e-05 | 1.028059e-04 | 1.680362e-04 | 1.631752e-04 | 1.925885e-04 | 1.505682e-04 | 1.607503e-04 | -1.062952e-05 | 3.495720e-07 | ... | -0.000008 | -8.776206e-06 | -8.885232e-08 | -0.000020 | 0.000001 | 2.022017e-05 | 9.856786e-07 | -1.273798e-07 | -0.000009 | -4.364141e-06 |
| A24ATI | -0.000136 | 1.815885e-07 | -1.970058e-05 | -8.950196e-06 | -6.468652e-06 | -5.006746e-06 | -4.471400e-06 | -1.062952e-05 | 2.284620e-03 | -8.700566e-06 | ... | -0.000054 | 2.621023e-04 | 8.558264e-05 | 0.000116 | 0.002265 | 1.269783e-03 | -3.005263e-05 | -1.886110e-04 | -0.000244 | -3.098541e-05 |
| CAPG211S | -0.000012 | -1.663361e-06 | -2.318434e-06 | -8.677502e-07 | -5.672107e-07 | -6.617963e-07 | -3.811469e-07 | 3.495720e-07 | -8.700566e-06 | 1.257110e-05 | ... | 0.000022 | 2.690789e-05 | 6.010570e-06 | 0.000007 | -0.000015 | -2.484340e-05 | -1.666452e-06 | -2.063607e-05 | -0.000020 | 7.926096e-07 |
| CAPUTLG211S | 0.000076 | 2.122961e-05 | 1.792528e-05 | -1.539373e-06 | -6.561594e-06 | -1.126550e-06 | -2.835937e-06 | -6.179695e-06 | -1.431268e-05 | 4.491472e-06 | ... | 0.000612 | -5.242073e-06 | 7.510665e-05 | 0.000001 | -0.000114 | 1.616584e-05 | 9.587596e-06 | -2.247655e-05 | -0.000195 | -1.518397e-05 |
| CPIENGSL | -0.000099 | -1.397815e-05 | -2.175044e-05 | 2.445855e-06 | 1.073625e-05 | 7.594321e-06 | 3.278638e-06 | 3.403180e-06 | 9.512141e-04 | -3.792738e-06 | ... | -0.000162 | 1.675363e-04 | 3.686100e-05 | 0.000067 | 0.001494 | 6.109216e-04 | -6.545729e-06 | -1.415337e-04 | -0.000085 | -1.125596e-05 |
| IPG21112S | 0.000097 | 2.852511e-05 | 2.243326e-05 | -4.750573e-06 | -1.089109e-05 | -3.644576e-06 | -5.670545e-06 | -8.290188e-06 | -5.428684e-05 | 2.178262e-05 | ... | 0.000853 | 6.687726e-06 | 1.049354e-04 | 0.000008 | -0.000266 | 2.466524e-06 | 3.072314e-06 | -9.356493e-05 | -0.000304 | -1.839977e-05 |
| IPN213111N | -0.000203 | -3.309475e-05 | -4.207374e-05 | 5.558130e-06 | -5.671556e-06 | -5.182760e-06 | 8.773246e-07 | -8.776206e-06 | 2.621023e-04 | 2.690789e-05 | ... | 0.000007 | 2.072267e-03 | 1.198737e-04 | 0.000092 | 0.000366 | -1.084505e-03 | -5.325756e-05 | -1.793491e-04 | -0.000408 | 6.963129e-06 |
| INDPRO | 0.000021 | 8.117655e-06 | 4.476718e-06 | 2.325853e-06 | 3.420240e-07 | 2.556412e-06 | 1.206808e-06 | -8.885232e-08 | 8.558264e-05 | 6.010570e-06 | ... | 0.000105 | 1.198737e-04 | 5.585615e-05 | 0.000012 | 0.000093 | 7.607697e-05 | 2.066817e-06 | 2.755146e-05 | -0.000031 | -4.072292e-06 |
| OVXCLS | -0.000304 | -4.301970e-05 | -5.634972e-05 | -8.571286e-06 | -1.216793e-05 | -1.871316e-05 | -1.114592e-05 | -1.957717e-05 | 1.162309e-04 | 6.868641e-06 | ... | 0.000008 | 9.247344e-05 | 1.169254e-05 | 0.002223 | 0.000230 | 1.584782e-05 | 1.031629e-05 | 5.812202e-05 | 0.000074 | 2.993011e-06 |
| PCU211211 | -0.000155 | -1.421436e-05 | -3.778895e-05 | -1.723010e-05 | -1.934046e-06 | -8.786978e-06 | -1.626545e-05 | 1.157952e-06 | 2.265204e-03 | -1.545715e-05 | ... | -0.000266 | 3.663526e-04 | 9.278279e-05 | 0.000230 | 0.005806 | 1.497650e-03 | 1.703032e-05 | -3.551215e-04 | -0.000139 | 1.117416e-05 |
| WTISPLC | 0.000263 | 8.061695e-05 | 4.253496e-05 | -2.175281e-06 | 2.613584e-07 | 1.443986e-05 | -7.584528e-06 | 2.022017e-05 | 1.269783e-03 | -2.484340e-05 | ... | 0.000002 | -1.084505e-03 | 7.607697e-05 | 0.000016 | 0.001498 | 7.757685e-03 | 1.282211e-04 | -3.952045e-04 | -0.000422 | -2.665936e-05 |
| PAPR_OPEC | 0.000024 | 5.934010e-06 | 5.893857e-06 | -1.338520e-06 | 9.041412e-07 | -3.348231e-07 | 7.432304e-07 | 9.856786e-07 | -3.005263e-05 | -1.666452e-06 | ... | 0.000003 | -5.325756e-05 | 2.066817e-06 | 0.000010 | 0.000017 | 1.282211e-04 | 1.075015e-04 | 6.249008e-05 | 0.000051 | 3.828361e-06 |
| RCRR01NUS_1 | -0.000060 | 5.148005e-06 | -1.297691e-05 | -9.445202e-06 | 1.618715e-06 | -1.665236e-05 | -1.546609e-05 | -1.273798e-07 | -1.886110e-04 | -2.063607e-05 | ... | -0.000094 | -1.793491e-04 | 2.755146e-05 | 0.000058 | -0.000355 | -3.952045e-04 | 6.249008e-05 | 6.248434e-03 | 0.003181 | 1.878966e-05 |
| RCRR01R48F_1 | -0.000180 | -3.283385e-05 | -3.682425e-05 | -7.563261e-06 | -1.223134e-05 | -1.355884e-05 | -3.026788e-06 | -9.075649e-06 | -2.438392e-04 | -1.983978e-05 | ... | -0.000304 | -4.077781e-04 | -3.130376e-05 | 0.000074 | -0.000139 | -4.215763e-04 | 5.145207e-05 | 3.181375e-03 | 0.003763 | 1.179445e-05 |
| WGFUPUS2 | -0.000017 | 1.519193e-06 | 3.804534e-07 | -1.877971e-06 | -6.157678e-06 | -7.137386e-06 | -8.471577e-06 | -4.364141e-06 | -3.098541e-05 | 7.926096e-07 | ... | -0.000018 | 6.963129e-06 | -4.072292e-06 | 0.000003 | 0.000011 | -2.665936e-05 | 3.828361e-06 | 1.878966e-05 | 0.000012 | 9.523259e-04 |
22 rows × 22 columns
Even though our covariances span multiple orders of magnitude, we will not normalize our data (beyond computing the percent change, which we already did) because the values are semantically meaningful: negative values reflect real-world declines in price, inventory, output, etcetera. Also, since the percent change of our economic data does not represent true returns in the financial sense, we will not de-annualize any of our data. Instead, we will require our models to handle the various orders of magnitude as they are.
In [10]:
sns.pairplot(train_df)
plt.show()
Changes in proved oil reserves appear to take on a small set of discrete values, while the stock prices tend to have more continuously distributed price changes.
In [11]:
corr = train_df.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
fig, ax = plt.subplots(1, 1, figsize=(10, 8))
cmap = sns.diverging_palette(230, 20, as_cmap=True)
sns.heatmap(
corr, mask=mask, cmap=cmap, center=0, annot=True,
fmt="0.2f", annot_kws={"size": 7}, square=True, linewidths=.5,
cbar_kws={"shrink": .75}, ax=ax
)
plt.show()
As expected, the stocks CVX and SHEL, along with the sector ETFs XLE and XOM all exhibit strong correlation with the general market index ^SPX, but more moderate correlation with oil futures BZ=F and CL=F. Meanwhile, the futures have strong correlation with each other. Oil/gas capacity utilization (CAPUTLG211S) and oil production (IPG21112S) are almost perfectly correlated, suggesting that we should examine the possibility of dropping one of them.
Closer Look: Utilization and Production¶
In [12]:
utilization_label = 'CAPUTLG211S'
production_label = 'IPG21112S'
corr = train_df[[utilization_label, production_label]].corr()
plt.title(f"Correlation: {corr.loc[utilization_label, production_label]:.2f}")
sns.scatterplot(train_df, x=utilization_label, y=production_label)
plt.show()
Let's remove the lower and upper extrema and replot:
In [13]:
cond1 = (train_df[utilization_label] != train_df[utilization_label].max())
cond2 = (train_df[utilization_label] != train_df[utilization_label].min())
corr = train_df[cond1 & cond2][[utilization_label, production_label]].corr()
plt.title(f"New Correlation: {corr.loc[utilization_label, production_label]:.2f}")
sns.scatterplot(train_df[cond1 & cond2], x=utilization_label, y=production_label)
plt.show()
Removing the upper/lower extrema reduces the correlation from 0.97 down to 0.90 -- which is still strong. When did these extrema occur?
In [14]:
train_df[~cond1].index, train_df[~cond2].index
Out[14]:
(DatetimeIndex(['2009-01-02', '2009-01-05', '2009-01-06', '2009-01-07',
'2009-01-08', '2009-01-09', '2009-01-12', '2009-01-13',
'2009-01-14', '2009-01-15', '2009-01-16', '2009-01-20',
'2009-01-21', '2009-01-22', '2009-01-23', '2009-01-26',
'2009-01-27', '2009-01-28', '2009-01-29', '2009-01-30'],
dtype='datetime64[ns]', name='Date', freq=None),
DatetimeIndex(['2008-12-01', '2008-12-02', '2008-12-03', '2008-12-04',
'2008-12-05', '2008-12-08', '2008-12-09', '2008-12-10',
'2008-12-11', '2008-12-12', '2008-12-15', '2008-12-16',
'2008-12-17', '2008-12-18', '2008-12-19', '2008-12-22',
'2008-12-23', '2008-12-24', '2008-12-26', '2008-12-29',
'2008-12-30', '2008-12-31'],
dtype='datetime64[ns]', name='Date', freq=None))
These extreme values appeared during the two worst months of the 2008 market crash (September and October, lagged by 3 months each). Those were extraordinary times, so this economic data appears to be a realistic and fair representation of what happened. For that reason, we will not remove these outliers from the data. Since the capacity utilization (CAPUTLG211S) also reflects natural gas, it does technically introduce additional data not visible in oil production (IPG21112S), which is likely the reason for the less-than perfect correlation. We could probably drop one of these series without trouble, but on the off-chance the non-mutual information is useful, we will keep them both and let the model decide which one is more important.
Combinatorial Purged Cross Validation¶
To improve our model development process and avoid overfitting our validation set, we will use combinatorial cross validation within our training set. To make sure our process is capable of predicting out-of-sample market dynamics for multiple years at a time, we will aim for segments of roughly 1 year each. Our training set spans 11.5 years, so we will divide it into 11 segments and pick 8 for training, with the remaining 3 for cross-validation. In every iteration, the model development process will train a model on ~8 years of data and predicting on the unseen 3 years. This arrangement will give us $\binom{11}{8}=165$ combinations, which is certainly enough to estimate the first and second statistical moments of our performance metrics.
In [15]:
class CPCV:
"""Keeps track of all Combinatorial Purged Cross Validation metadata. Provides
quality-of-life methods for accessing CPCV indices and data.
"""
def __init__(
self,
train_df: pd.DataFrame,
n_segments: int=8,
k_choose: int=6,
purge_horizon: int=1,
show: bool=False
) -> None:
self.train_df = train_df
self.n_segments = n_segments
self.k_choose = k_choose
self.purge_horizon = purge_horizon
# Split training set into n_groups of roughly equal size
self.n_train = train_df.shape[0]
dn = self.n_train // self.n_segments
x = [dn * i for i in range(self.n_segments + 1)]
x[-1] = self.n_train
# Create index slices for each segment
self.segments = [np.arange(a, b, dtype=int) for a, b in zip(x[:-1], x[1:])]
# Create groups comprised of various combinations of segments
self.groups = list(itertools.combinations(self.segments, self.k_choose))
self.n_groups = len(self.groups)
group_idxs = [np.r_[group] for group in self.groups]
print(f"{len(group_idxs)} CPCV sets")
if show:
display(group_idxs)
def get_group_idxs(self, combo_idx: int) -> dict:
"""For a given combo_idx, returns the generated indices for the cpcv_train,
cpcv_val, and purged sets. These indices can be used to slice out the desired
data from self.train_df.
"""
# Get values in ar1 that are not in ar2; that is, get idxs that are not in
# train and use them in val.
ar1 = np.arange(self.n_train, dtype=np.int64)
ar2 = np.r_[self.groups[combo_idx]]
val_idxs = np.setdiff1d(ar1=ar1, ar2=ar2)
# Now purge last purge_horizon indices from each train segment
train_idxs_seperated = [x[:-self.purge_horizon] for x in self.groups[combo_idx]]
train_idxs = np.r_[tuple(train_idxs_seperated)]
# Keep track of the purged indices for verification
purged_idxs_seperated = [x[-self.purge_horizon:] for x in self.groups[combo_idx]]
purged_idxs = np.r_[tuple(purged_idxs_seperated)]
result = {
'cpcv_train_idxs': train_idxs,
'cpcv_val_idxs': val_idxs,
'purged_idxs': purged_idxs
}
return result
def get_cpcv_sets(self, combo_idx: int) -> (pd.DataFrame, pd.DataFrame):
"""Returns the CPCV training, validation, and purged set data (not indices),
checking that the total number of elements matches the expected length.
"""
group_idxs = self.get_group_idxs(combo_idx=combo_idx)
cpcv_train = self.train_df.iloc[group_idxs['cpcv_train_idxs']]
cpcv_val = self.train_df.iloc[group_idxs['cpcv_val_idxs']]
cpcv_purged = self.train_df.iloc[group_idxs['purged_idxs']]
assert cpcv_val.shape[0] + cpcv_train.shape[0] + cpcv_purged.shape[0] == self.train_df.shape[0]
result = {
'cpcv_train': cpcv_train,
'cpcv_val': cpcv_val,
'cpcv_purged': cpcv_purged,
}
return result
cpcv = CPCV(train_df=train_df, n_segments=11, k_choose=8, purge_horizon=TARGET_HORIZON_DAYS, show=True)
print()
i = 0
cpcv_idxs = cpcv.get_group_idxs(combo_idx=i)
cpcv_data = cpcv.get_cpcv_sets(combo_idx=i)
cpcv_train = cpcv_data['cpcv_train']
cpcv_val = cpcv_data['cpcv_val']
print(f"CPCV Train shape for i={i}: {cpcv_train.shape}")
print(f"CPCV Val shape for i={i}: {cpcv_val.shape}")
print(f"CPCV Purged shape for i={i}: {cpcv_data['cpcv_purged'].shape}")
165 CPCV sets
[array([ 0, 1, 2, ..., 2085, 2086, 2087], shape=(2088,)), array([ 0, 1, 2, ..., 2346, 2347, 2348], shape=(2088,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2346, 2347, 2348], shape=(2088,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2346, 2347, 2348], shape=(2088,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2346, 2347, 2348], shape=(2088,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2346, 2347, 2348], shape=(2088,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2346, 2347, 2348], shape=(2088,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2346, 2347, 2348], shape=(2088,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2607, 2608, 2609], shape=(2088,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 0, 1, 2, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2346, 2347, 2348], shape=(2088,)), array([ 261, 262, 263, ..., 2607, 2608, 2609], shape=(2088,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2607, 2608, 2609], shape=(2088,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2607, 2608, 2609], shape=(2088,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2607, 2608, 2609], shape=(2088,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2607, 2608, 2609], shape=(2088,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2607, 2608, 2609], shape=(2088,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2607, 2608, 2609], shape=(2088,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 261, 262, 263, ..., 2872, 2873, 2874], shape=(2092,)), array([ 522, 523, 524, ..., 2607, 2608, 2609], shape=(2088,)), array([ 522, 523, 524, ..., 2872, 2873, 2874], shape=(2092,)), array([ 522, 523, 524, ..., 2872, 2873, 2874], shape=(2092,)), array([ 522, 523, 524, ..., 2872, 2873, 2874], shape=(2092,)), array([ 522, 523, 524, ..., 2872, 2873, 2874], shape=(2092,)), array([ 522, 523, 524, ..., 2872, 2873, 2874], shape=(2092,)), array([ 522, 523, 524, ..., 2872, 2873, 2874], shape=(2092,)), array([ 522, 523, 524, ..., 2872, 2873, 2874], shape=(2092,)), array([ 783, 784, 785, ..., 2872, 2873, 2874], shape=(2092,))]
CPCV Train shape for i=0: (2048, 22) CPCV Val shape for i=0: (787, 22) CPCV Purged shape for i=0: (40, 22)
Note that each element will occur in $\binom{n - 1}{k - 1}$ of the combinations, so we would expect to see the first segment appear in 120 out of 165 combinations given n=11 and k=8.
Single Model Development¶
Discretize Data¶
In [16]:
class SimpleDiscretizer:
"""Discretizes data by sign. 0 means negative, 1 means positive.
This class exists for the sake of consistency: we may want to
use more complicated discretizers in the future, but with similar
usage patterns.
"""
def __init__(self) -> None:
pass
def fit(self, df: pd.DataFrame) -> None:
pass
def transform(self, df: pd.DataFrame) -> pd.DataFrame:
"""Applies the sign-based discretization."""
return (df > 0) * 1
##########################################################################################
### Example: fit discretizer to training set, use model to transform validation set
##########################################################################################
discretizer = SimpleDiscretizer()
discretizer.fit(cpcv_train)
cpcv_train_discrete = discretizer.transform(cpcv_train)
cpcv_val_discrete = discretizer.transform(cpcv_val)
cpcv_val_discrete
Out[16]:
| target | BZ=F | CL=F | CVX | SHEL | XLE | XOM | ^SPX | A24ATI | CAPG211S | ... | IPG21112S | IPN213111N | INDPRO | OVXCLS | PCU211211 | WTISPLC | PAPR_OPEC | RCRR01NUS_1 | RCRR01R48F_1 | WGFUPUS2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Date | |||||||||||||||||||||
| 2015-11-13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | ... | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
| 2015-11-16 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | ... | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 |
| 2015-11-17 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | ... | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 |
| 2015-11-18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | ... | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 |
| 2015-11-19 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | ... | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 2018-12-24 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | ... | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 |
| 2018-12-26 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | ... | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 |
| 2018-12-27 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | ... | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 |
| 2018-12-28 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | ... | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 |
| 2018-12-31 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | ... | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 0 |
787 rows × 22 columns
Encoding Market Regimes¶
Labeling market regimes is challenging because there is no single "right answer." Our goal is train an HMM that can identify when markets are generally trending upward, downward, or sideways, under the hypothesis that such information can be used to predict price movements in the near future. We don't have to be perfectly accurate; we just need enough of a statistical edge to build a profitable trading strategy.
We will initialize the HMM under the assumption that there are three market regimes (the HMM's "hidden states"): bulls, bears, and stagnant markets. We will assume that each market emits a different proportion of positive/negative price movements. More specifically, we will start with the assumption that bull markets are more likely to emit positive price movements than negative movements, and vice versa for bear markets; for stagnant markets, we will assume equal probabilities of positive and negative price movements.
In [17]:
class HMMEncoder:
"""Container for all HMM models trained on the allowed features
of the data. Provides methods for fitting and encoding market regimes.
"""
def __init__(
self,
allowed_features: list,
hidden_state_emission_priors: list,
transition_priors: list,
start_priors: list,
max_iter: int=100,
) -> None:
self.allowed_features = allowed_features
self.hidden_state_emission_priors = hidden_state_emission_priors
self.transition_priors = transition_priors
self.start_priors = start_priors
self.max_iter = max_iter
self.models = {}
def expand_priors(self, prior_list: list, series: pd.Series) -> list:
"""Expands a priors array of size N to an array of size M, where M > N and M % N == 0.
Each of the N segments of M adds up to their corresponding value from N, and the sum of M
equals the sum of N. This makes it easier to handle configurations where some columns are
discretized by quantile and others aren't.
"""
new_prior_list = []
for prior_object in prior_list:
priors = prior_object.probs[0].numpy()
n_priors = len(priors)
n_unique = len(series.unique())
if n_unique == n_priors:
new_prior_list.append(prior_object)
else:
assert n_unique % n_priors == 0
n_per_segment = int(n_unique / n_priors)
new_priors = []
for i in range(n_priors):
val = priors[i] / n_per_segment
new_priors = new_priors + [val] * n_per_segment
new_prior_list.append(Categorical([new_priors]))
return new_prior_list
def fit(
self,
df: pd.DataFrame,
verbose: int=0
) -> None:
"""Trains a new HMM on each allowed feature in df. Stores the HMM
object references in a dictionary.
"""
for feature in df.columns:
if feature in self.allowed_features:
verb = False
if verbose > 0:
print(f"Fitting {feature}...")
if verbose == 2:
verb = True
expanded_emission_priors = self.expand_priors(
prior_list=self.hidden_state_emission_priors,
series=df[feature]
)
self.models[feature] = DenseHMM(
expanded_emission_priors,
edges=self.transition_priors,
starts=self.start_priors,
verbose=verb,
max_iter=self.max_iter
)
X = df[feature].values.reshape(1, -1, 1)
self.models[feature].fit(X)
def encode(
self,
df: pd.DataFrame
) -> pd.DataFrame:
"""Uses previously trained HMMs to encode the columns of df as integers
representing market regimes.
"""
data = {}
for feature in df.columns:
if feature in self.allowed_features: # Encode feature as market regime
X = df[feature].values.reshape(1, -1, 1)
pred = np.argmax(self.models[feature].predict_proba(X).numpy()[0], axis=1)
data[feature] = pred
else: # Use feature as-is
data[feature] = df[feature]
return pd.DataFrame(data)
##########################################################################################
### Example: fit regime detector to training set, use model to transform validation set
##########################################################################################
# Prior distributions for emissions
hs_bear = Categorical([[0.6, 0.4]])
hs_stag = Categorical([[0.5, 0.5]])
hs_bull = Categorical([[0.4, 0.6]])
hidden_states = [hs_bear, hs_stag, hs_bull]
# Prior distributions for HS transitions
edges = [
[0.90, 0.05, 0.05],
[0.05, 0.90, 0.05],
[0.05, 0.05, 0.90]
]
# Prior distributions for starting state
starts = [0.3, 0.3, 0.4]
features_to_encode = yf_features
hmm = HMMEncoder(
allowed_features=features_to_encode,
hidden_state_emission_priors=hidden_states,
transition_priors=edges,
start_priors=starts,
)
hmm.fit(cpcv_train_discrete, verbose=1)
cpcv_train_encoded = hmm.encode(cpcv_train_discrete)
cpcv_val_encoded = hmm.encode(cpcv_val_discrete)
cpcv_val_encoded
Fitting BZ=F... Fitting CL=F... Fitting CVX... Fitting SHEL... Fitting XLE... Fitting XOM... Fitting ^SPX...
Out[17]:
| target | BZ=F | CL=F | CVX | SHEL | XLE | XOM | ^SPX | A24ATI | CAPG211S | ... | IPG21112S | IPN213111N | INDPRO | OVXCLS | PCU211211 | WTISPLC | PAPR_OPEC | RCRR01NUS_1 | RCRR01R48F_1 | WGFUPUS2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Date | |||||||||||||||||||||
| 2015-11-13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 1 | ... | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
| 2015-11-16 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 1 | ... | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 |
| 2015-11-17 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 1 | ... | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 |
| 2015-11-18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | ... | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 |
| 2015-11-19 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 1 | ... | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 2018-12-24 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | ... | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 |
| 2018-12-26 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | ... | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 |
| 2018-12-27 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | ... | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 |
| 2018-12-28 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | ... | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 |
| 2018-12-31 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | ... | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 0 |
787 rows × 22 columns
Visualize HMM Regime Encodings¶
In [18]:
dataset = cpcv_val_encoded
dt_idxs = dataset.index
for feature in features_to_encode:
price_data = yf_df.loc[dt_idxs, feature]
c_dict = {0: 'red', 1: 'blue', 2: 'green'}
hs_dict = {0: 'bear', 1: 'stag', 2: 'bull'}
plt.figure(figsize=(12, 6))
plt.plot(price_data, label='price')
for val in c_dict.keys():
x = dataset[dataset[feature]==val][feature].index
y = price_data.loc[x]
plt.scatter(x, y, c=c_dict[val], s=5, label=hs_dict[val])
plt.title(f"Regime Encoding for {feature}")
plt.legend()
plt.show()
The HMM regime-encoding process is more of an art than a science, as we can see from some of the questionable regime labels applied in the graphs above. Even so, if the HMM is able to represent trends better than the day-to-day price changes, that information could potentially be enough to predict WIT price movements with a statistical edge.
Structure-Fitting¶
In [19]:
def verify_all_nodes_connected_to_graph(model):
"""Checks that there are no solitary, disconnectd nodes in the model. Warns
the user of any disconnected nodes that may cause issues during inference.
"""
connected_nodes = set([str(s) for s in np.array(model.edges).flatten()])
disconnected_nodes = []
for feature in list(model.nodes):
if feature not in connected_nodes:
disconnected_nodes.append(feature)
if len(disconnected_nodes) != 0:
warnings.warn(f"Disconnected nodes: {disconnected_nodes}")
def enforce_states_are_represented(
model: DiscreteBayesianNetwork,
hidden_states: list,
) -> DiscreteBayesianNetwork:
"""Verifies that all possible states are represented in the Bayesian
Network's conditional probability tables. Sometimes the training data
does not fully represent all possible states. In these cases, the
Bayesian Network will be surprised if it sees a state that wasn't in
the training data. To avoid such surprises, this function initializes
uniform CPDs for each feature according to the expected cardinality
of that feature.
"""
n_hidden_states = len(hidden_states)
n_emissions = len(hidden_states[0].probs[0])
for feature in list(model.nodes):
if feature in features_to_encode:
cpd = TabularCPD(
variable=feature,
variable_card=n_hidden_states,
values=[[1./n_hidden_states] for i in range(n_hidden_states)]
)
else:
cpd = TabularCPD(
variable=feature,
variable_card=n_emissions,
values=[[1./n_emissions] for i in range(n_emissions)]
)
model.add_cpds(cpd)
return model
In [20]:
max_indegree = 4
data = cpcv_train_encoded
hc = HillClimbSearch(data=data)
expert = DiscreteBayesianNetwork()
expert.add_nodes_from(data.columns)
expert.add_edges_from([
('^SPX', 'XLE'), # Market influences the sector in a top-down fashion
('PCU211211', 'CPIENGSL'), # Producer price probably influences consumer prices
('IPN213111N', 'CAPG211S'), # Drilling influences capacity
])
start = time.time()
model = hc.estimate(
scoring_method=K2(data=data),
start_dag=expert,
max_indegree=max_indegree
)
stop = time.time()
print(f"Time: {(stop-start):.2f} [sec]")
verify_all_nodes_connected_to_graph(model)
0%| | 0/1000000 [00:00<?, ?it/s]
Time: 0.91 [sec]
In [21]:
len(model.edges()), len(model.nodes()) * max_indegree, len(model.nodes())
Out[21]:
(65, 88, 22)
In [22]:
G = nx.DiGraph()
G.add_edges_from(model.edges())
pos = nx.spring_layout(G, k=5)
plt.figure(figsize=(10,8))
nx.draw(G, pos, with_labels=True, node_size=4000)
plt.show()
Parameter-Fitting¶
In [23]:
def handle_missing_cpds(model, data):
"""If any node is missing a conditional probability table (CPD), this
function will create a CPD for that node using a uniform distribution
over the domain represented in the data. This helps to avoid problems
during inference.
"""
nodes = list(model.nodes)
for node in nodes:
if not model.get_cpds(node):
cardinality = len(data[node].unique())
values = np.ones(shape=(cardinality, 1)) / cardinality
cpd = TabularCPD(variable=node, variable_card=cardinality, values=values)
model.add_cpds(cpd)
def validate_cpds(model):
"""Raises an error if any nodes are missing a CPD. This should be called
after handle_missing_cpds() as a safeguard.
"""
missing = [node for node in model.nodes() if not model.get_cpds(node)]
if missing:
raise ValueError(f"Missing CPDs for: {missing}")
In [24]:
start = time.time()
model.fit(
cpcv_train_encoded,
estimator=BayesianEstimator,
prior_type="K2"
)
handle_missing_cpds(model, data)
stop = time.time()
print(f"Time: {(stop-start):.2f} [sec]")
validate_cpds(model)
Time: 0.03 [sec]
In [25]:
pred_labels = [TARGET_LABEL]
dataset = cpcv_train_encoded
y_pred = model.predict(dataset.drop(pred_labels, axis=1))[pred_labels]
y_true = dataset[pred_labels]
accuracy = accuracy_score(y_true=y_true, y_pred=y_pred)
0%| | 0/1301 [00:00<?, ?it/s]
In [26]:
cm = confusion_matrix(y_true, y_pred)
disp = ConfusionMatrixDisplay(confusion_matrix=cm)
disp.plot(cmap='Blues')
plt.show()
print(f"Row sums (True Labels): {cm.sum(axis=1)}")
print(f"Column sums (Predicted Labels): {cm.sum(axis=0)}")
print(f"Accuracy: {accuracy:.3f}")
print(f"Precision: {precision_score(y_true=y_true, y_pred=y_pred):.3f}")
print(f"Recall: {recall_score(y_true=y_true, y_pred=y_pred):.3f}")
print(f"F1 Score: {f1_score(y_true=y_true, y_pred=y_pred):.3f}")
print(f"P(pred=0): {(1 - y_pred).mean()[0]:.3f}")
print(f"P(pred=1): {y_pred.mean()[0]:.3f}")
print(f"P(true=0): {(1 - y_true).mean()[0]:.3f}")
print(f"P(true=1): {y_true.mean()[0]:.3f}")
Row sums (True Labels): [ 999 1049] Column sums (Predicted Labels): [1124 924] Accuracy: 0.708 Precision: 0.745 Recall: 0.656 F1 Score: 0.697 P(pred=0): 0.549 P(pred=1): 0.451 P(true=0): 0.488 P(true=1): 0.512
Predicting on CPCV Val¶
In [27]:
pred_labels = [TARGET_LABEL]
dataset = cpcv_val_encoded
y_pred = model.predict(dataset.drop(pred_labels, axis=1))[pred_labels]
y_true = dataset[pred_labels]
accuracy = accuracy_score(y_true=y_true, y_pred=y_pred)
0%| | 0/497 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
In [28]:
cm = confusion_matrix(y_true, y_pred)
disp = ConfusionMatrixDisplay(confusion_matrix=cm)
disp.plot(cmap='Blues')
plt.show()
print(f"Row sums (True Labels): {cm.sum(axis=1)}")
print(f"Column sums (Predicted Labels): {cm.sum(axis=0)}")
print(f"Accuracy: {accuracy:.3f}")
print(f"Precision: {precision_score(y_true=y_true, y_pred=y_pred):.3f}")
print(f"Recall: {recall_score(y_true=y_true, y_pred=y_pred):.3f}")
print(f"F1 Score: {f1_score(y_true=y_true, y_pred=y_pred):.3f}")
print(f"P(pred=0): {(1 - y_pred).mean()[0]:.3f}")
print(f"P(pred=1): {y_pred.mean()[0]:.3f}")
print(f"P(true=0): {(1 - y_true).mean()[0]:.3f}")
print(f"P(true=1): {y_true.mean()[0]:.3f}")
Row sums (True Labels): [379 408] Column sums (Predicted Labels): [327 460] Accuracy: 0.682 Precision: 0.672 Recall: 0.757 F1 Score: 0.712 P(pred=0): 0.416 P(pred=1): 0.584 P(true=0): 0.482 P(true=1): 0.518
Model Development Pipeline¶
In [29]:
class ModelDevelopmentPipeline:
"""Executes the entire model development process:
1. Discretization (Train + Val)
2. Parameter Fitting for the HMM (Train)
2. Regime Encoding via HMM (Train + Val)
3. Structure Fitting for the Bayesian Network (Train)
4. Parameter Fitting for the Bayesian Network (Train)
5. Performance Evaluation (Val)
Returns the model and performance metrics.
"""
def __init__(
self,
features_to_encode: list,
hmm_verbosity: int,
edges: list,
hidden_states: list,
starts: list,
max_indegree: int,
expert_edges: list,
pred_labels: list,
return_model: bool=False
) -> None:
self.features_to_encode = features_to_encode
self.hmm_verbosity = hmm_verbosity
self.edges = edges
self.hidden_states = hidden_states
self.starts = starts
self.max_indegree = max_indegree
self.expert_edges = expert_edges
self.pred_labels = pred_labels
self.return_model = return_model
self.train_discrete = None
self.hmm = None
self.model = None
self.discretizer = SimpleDiscretizer()
def fit(self, train_dataset: pd.DataFrame) -> None:
"""Fits the discretizer, HMM, BN structure, and BN parameters to
train_dataset, storing each of these models internally.
"""
# Fit Discretizer
self.discretizer.fit(train_dataset)
train_discrete = self.discretizer.transform(train_dataset)
# Fit HMM
self.hmm = HMMEncoder(
allowed_features=self.features_to_encode,
hidden_state_emission_priors=self.hidden_states,
transition_priors=self.edges,
start_priors=self.starts,
)
self.hmm.fit(train_discrete, verbose=self.hmm_verbosity)
train_encoded = self.hmm.encode(train_discrete)
# Fit BN Structure
hc = HillClimbSearch(data=train_encoded)
expert = DiscreteBayesianNetwork()
expert.add_nodes_from(train_encoded.columns)
expert.add_edges_from(self.expert_edges)
self.model = hc.estimate(
scoring_method=K2(data=train_encoded),
start_dag=expert,
max_indegree=self.max_indegree
)
verify_all_nodes_connected_to_graph(self.model)
# Fit BN Parameters
self.model.fit(
train_encoded,
estimator=BayesianEstimator,
prior_type="K2"
)
handle_missing_cpds(self.model, train_encoded)
validate_cpds(self.model)
def predict(
self,
pred_dataset: pd.DataFrame,
pred_labels: list,
) -> dict:
"""Applies the discretizer, HMM encoder, and BN inference to predict
pred_labels over pred_dataset.
"""
pred_discrete = self.discretizer.transform(pred_dataset)
pred_encoded = self.hmm.encode(pred_discrete)
y_pred = self.model.predict(pred_encoded.drop(pred_labels, axis=1))[pred_labels]
return y_pred
def evaluate(
self,
pred_dataset: pd.DataFrame,
pred_labels: list,
) -> dict:
"""Returns performance statistics for classification."""
pred_discrete = self.discretizer.transform(pred_dataset)
pred_encoded = self.hmm.encode(pred_discrete)
y_pred = self.model.predict(pred_encoded.drop(pred_labels, axis=1))[pred_labels]
y_true = pred_encoded[pred_labels]
accuracy = accuracy_score(y_true=y_true, y_pred=y_pred)
precision = precision_score(y_true=y_true, y_pred=y_pred)
recall = recall_score(y_true=y_true, y_pred=y_pred)
f1 = f1_score(y_true=y_true, y_pred=y_pred)
performance_stats = {
'accuracy': accuracy,
'precision': precision,
'recall': recall,
'f1': f1,
}
return performance_stats
In [30]:
####################################################################################
# Define Settings
####################################################################################
# Settings for encoder
features_to_encode = yf_features
# Settings for HMM: hidden state transitions, emissions, and starts
hmm_verbosity = 0
edges = [
[0.90, 0.05, 0.05],
[0.05, 0.90, 0.05],
[0.05, 0.05, 0.90]
]
hs_bear = Categorical([[0.6, 0.4]])
hs_stag = Categorical([[0.5, 0.5]])
hs_bull = Categorical([[0.4, 0.6]])
hidden_states = [hs_bear, hs_stag, hs_bull]
starts = [0.3, 0.3, 0.4]
# Settings for Hill Climb Search
max_indegree = 4
expert_edges = [
('^SPX', 'XLE'), # Market influences the sector in a top-down fashion
('PCU211211', 'CPIENGSL'), # Producer price probably influences consumer prices
('IPN213111N', 'CAPG211S'), # Drilling influences capacity
]
# Settings for prediction
pred_labels = [TARGET_LABEL]
# Settings for returned values
return_model = False
####################################################################################
# Iterate Through Combinatorial Cross Validation (CPCV) Train and Val Sets
####################################################################################
performance_results = []
markov_blanket_nodes = []
for i in range(cpcv.n_groups):
cpcv_idxs = cpcv.get_group_idxs(combo_idx=i)
cpcv_data = cpcv.get_cpcv_sets(combo_idx=i)
cpcv_train = cpcv_data['cpcv_train']
cpcv_val = cpcv_data['cpcv_val']
mdp = ModelDevelopmentPipeline(
features_to_encode=features_to_encode,
hmm_verbosity=hmm_verbosity,
edges=edges,
hidden_states=hidden_states,
starts=starts,
max_indegree=max_indegree,
expert_edges=expert_edges,
pred_labels=pred_labels,
return_model=return_model
)
mdp.fit(train_dataset=cpcv_train)
single_performance_result = mdp.evaluate(pred_dataset=cpcv_val, pred_labels=pred_labels)
performance_results.append(single_performance_result)
markov_blanket_nodes = markov_blanket_nodes + mdp.model.get_markov_blanket(TARGET_LABEL)
performance_results_cpcv = pd.DataFrame(performance_results)
performance_results_cpcv
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/497 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/484 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/452 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/482 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/483 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/486 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/500 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/464 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/486 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/452 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/510 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/502 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/512 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/449 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/487 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/446 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/497 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/506 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/479 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/468 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/485 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/470 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/497 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/461 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/501 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/438 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/489 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/495 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/470 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/440 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/511 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/509 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/490 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/448 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/475 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/499 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/474 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/498 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/480 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/496 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/450 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/494 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/508 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/472 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/442 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/493 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/504 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/516 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/474 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/480 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/474 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/496 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/481 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/437 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/501 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/482 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/502 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/478 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/504 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/462 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/461 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/427 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/483 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/504 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/467 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/441 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/505 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/522 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/500 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/457 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/455 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/455 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/469 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/449 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/402 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/417 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/423 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/433 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/461 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/438 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/395 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/407 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/435 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/403 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/428 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/418 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/444 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/401 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/395 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/387 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/414 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/425 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/408 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/371 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/455 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/482 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/454 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/438 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/430 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/430 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/429 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/412 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/372 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/398 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/416 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/412 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
/var/folders/57/nmtlyvms16b9m26k6y703_000000gn/T/ipykernel_92734/226775345.py:12: UserWarning: Disconnected nodes: ['WGFUPUS2']
warnings.warn(f"Disconnected nodes: {disconnected_nodes}")
0%| | 0/440 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/408 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/378 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/411 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/428 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/402 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/417 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
/var/folders/57/nmtlyvms16b9m26k6y703_000000gn/T/ipykernel_92734/226775345.py:12: UserWarning: Disconnected nodes: ['WGFUPUS2']
warnings.warn(f"Disconnected nodes: {disconnected_nodes}")
0%| | 0/436 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/411 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/386 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/399 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/417 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/406 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
/var/folders/57/nmtlyvms16b9m26k6y703_000000gn/T/ipykernel_92734/226775345.py:12: UserWarning: Disconnected nodes: ['WGFUPUS2']
warnings.warn(f"Disconnected nodes: {disconnected_nodes}")
0%| | 0/423 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/424 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/421 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/432 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/396 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/413 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/385 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/435 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/501 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/486 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/454 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/501 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/508 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/503 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/450 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/498 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/501 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/500 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/452 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/451 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/486 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/415 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/481 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/381 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/387 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/370 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/459 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/394 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/380 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/394 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
/var/folders/57/nmtlyvms16b9m26k6y703_000000gn/T/ipykernel_92734/226775345.py:12: UserWarning: Disconnected nodes: ['WGFUPUS2']
warnings.warn(f"Disconnected nodes: {disconnected_nodes}")
0%| | 0/400 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/394 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/360 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/375 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/397 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/375 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/379 [00:00<?, ?it/s]
WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers WARNING:pgmpy:Found unknown state name. Trying to switch to using all state names as state numbers
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/386 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
/var/folders/57/nmtlyvms16b9m26k6y703_000000gn/T/ipykernel_92734/226775345.py:12: UserWarning: Disconnected nodes: ['WGFUPUS2']
warnings.warn(f"Disconnected nodes: {disconnected_nodes}")
0%| | 0/522 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/385 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/360 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/387 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/391 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/378 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/384 [00:00<?, ?it/s]
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/393 [00:00<?, ?it/s]
Out[30]:
| accuracy | precision | recall | f1 | |
|---|---|---|---|---|
| 0 | 0.682338 | 0.671739 | 0.757353 | 0.711982 |
| 1 | 0.688691 | 0.660147 | 0.718085 | 0.687898 |
| 2 | 0.678526 | 0.721014 | 0.530667 | 0.611367 |
| 3 | 0.708812 | 0.722222 | 0.650000 | 0.684211 |
| 4 | 0.667090 | 0.633554 | 0.749347 | 0.686603 |
| ... | ... | ... | ... | ... |
| 160 | 0.710089 | 0.754545 | 0.630380 | 0.686897 |
| 161 | 0.701149 | 0.748092 | 0.685315 | 0.715328 |
| 162 | 0.699872 | 0.705336 | 0.737864 | 0.721234 |
| 163 | 0.720307 | 0.760391 | 0.719907 | 0.739596 |
| 164 | 0.694764 | 0.785047 | 0.597156 | 0.678331 |
165 rows × 4 columns
CPCV Evaluation: Distribution of Performance Metrics¶
In [31]:
x = (train_df[TARGET_LABEL] > 0) * 1
print(f"Train Occurences of Target=1: {x.sum()}")
print(f"Train Occurences of Target=0: {(1 - x).sum()}")
print(f"Train P(Target=1) = {x.mean():.2f}")
print()
performance_results_cpcv.hist(bins=np.linspace(0., 1., 51))
plt.suptitle(f"N = {cpcv.n_groups}")
plt.show()
display(performance_results_cpcv.describe())
Train Occurences of Target=1: 1471 Train Occurences of Target=0: 1404 Train P(Target=1) = 0.51
| accuracy | precision | recall | f1 | |
|---|---|---|---|---|
| count | 165.000000 | 165.000000 | 165.000000 | 165.000000 |
| mean | 0.686972 | 0.706941 | 0.671729 | 0.684084 |
| std | 0.016155 | 0.038043 | 0.085747 | 0.037543 |
| min | 0.638570 | 0.606481 | 0.451977 | 0.557491 |
| 25% | 0.676884 | 0.678571 | 0.607427 | 0.658046 |
| 50% | 0.688378 | 0.709040 | 0.669118 | 0.686897 |
| 75% | 0.698856 | 0.730337 | 0.731183 | 0.714785 |
| max | 0.722861 | 0.807531 | 0.907500 | 0.755463 |
At first glance, these results look promising. The accuracy metrics are distributed around 68.6% with a standard deviation of 1.9%, suggesting the our HMM/BN approach is able to make correct predictions more often than not. Keep in mind that these accuracy measurements are for each model's predictions on its respective CPCV Validation set after training on the CPCV Training set, so they are a decent estimate of out-of-sample performance. Now, let's take a look at which features tend to appear most often in the Markov blanket for our target feature.
In [32]:
# Count the occurrences of each label
node_counts = Counter(markov_blanket_nodes)
sorted_nodes = sorted(node_counts.items(), key=lambda item: item[1], reverse=True)
# Separate labels and counts for plotting
ordered_nodes = [label for label, count in sorted_nodes]
ordered_pcts = [count / cpcv.n_groups for label, count in sorted_nodes]
excluded = [label for label in train_df.columns if label not in ordered_nodes and label != TARGET_LABEL]
print(f"Features with 0 occurences:")
print(excluded)
print()
# Create the histogram (bar plot)
plt.figure(figsize=(8, 6)) # Adjust figure size as needed
plt.bar(ordered_nodes, ordered_pcts, color='skyblue')
plt.xlabel("Feature")
plt.ylabel("Fraction of CPCV Iterations")
plt.title("Ordered Histogram of Feature Appearance in Markov Blanket of Target Feature")
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()
excluded = [
label for label in train_df.columns
if (label not in markov_blanket_nodes) and (label != TARGET_LABEL)
]
pd.DataFrame({'feature': ordered_nodes, '% occurence': ordered_pcts})
Features with 0 occurences: []
Out[32]:
| feature | % occurence | |
|---|---|---|
| 0 | CL=F | 1.000000 |
| 1 | OVXCLS | 0.963636 |
| 2 | BZ=F | 0.800000 |
| 3 | CVX | 0.478788 |
| 4 | RCRR01R48F_1 | 0.381818 |
| 5 | IPG21112S | 0.333333 |
| 6 | XOM | 0.333333 |
| 7 | CAPG211S | 0.248485 |
| 8 | IPN213111N | 0.206061 |
| 9 | XLE | 0.206061 |
| 10 | ^SPX | 0.151515 |
| 11 | SHEL | 0.127273 |
| 12 | A24ATI | 0.121212 |
| 13 | RCRR01NUS_1 | 0.084848 |
| 14 | CAPUTLG211S | 0.066667 |
| 15 | PCU211211 | 0.060606 |
| 16 | INDPRO | 0.042424 |
| 17 | WTISPLC | 0.042424 |
| 18 | CPIENGSL | 0.012121 |
| 19 | PAPR_OPEC | 0.012121 |
| 20 | WGFUPUS2 | 0.006061 |
Of all the features involved, the features that are most often included in the target's Markov blanket -- the set of nodes that directly influence predictions on the target feature -- are CL=F, OVXCLS, BZ=F, RCRR01R48F_1, and CVX. On the flip side, PAPR_OPEC appears in only 1.2% of all the Bayesian Networks trained, suggesting it has limited usefulness in predicting oil price movements.
Performance on Validation Set¶
Now that we've seen our model development process perform acceptably well in Combinatorial Purged Cross Validation, we will retrain the model on the entire Train set and evaluate its performance on the Validation set for the first time.
In [33]:
####################################################################################
# Define Settings
####################################################################################
# Settings for encoder
features_to_encode = yf_features
# Settings for HMM: hidden state transitions, emissions, and starts
hmm_verbosity = 0
edges = [
[0.90, 0.05, 0.05],
[0.05, 0.90, 0.05],
[0.05, 0.05, 0.90]
]
hs_bear = Categorical([[0.6, 0.4]])
hs_stag = Categorical([[0.5, 0.5]])
hs_bull = Categorical([[0.4, 0.6]])
hidden_states = [hs_bear, hs_stag, hs_bull]
starts = [0.3, 0.3, 0.4]
# Settings for Hill Climb Search
max_indegree = 4
expert_edges = [
('^SPX', 'XLE'), # Market influences the sector in a top-down fashion
('PCU211211', 'CPIENGSL'), # Producer price probably influences consumer prices
('IPN213111N', 'CAPG211S'), # Drilling influences capacity
]
# Settings for prediction
pred_labels = [TARGET_LABEL]
# Settings for returned values
return_model = True
####################################################################################
# Iterate Through Combinatorial Cross Validation (CPCV) Train and Val Sets
####################################################################################
mdp = ModelDevelopmentPipeline(
features_to_encode=features_to_encode,
hmm_verbosity=hmm_verbosity,
edges=edges,
hidden_states=hidden_states,
starts=starts,
max_indegree=max_indegree,
expert_edges=expert_edges,
pred_labels=pred_labels,
return_model=return_model
)
mdp.fit(train_dataset=train_df.iloc[:-TARGET_HORIZON_DAYS])
single_performance_result = mdp.evaluate(pred_dataset=val_df, pred_labels=pred_labels)
y_true = mdp.discretizer.transform(val_df[pred_labels])[TARGET_LABEL]
y_pred = mdp.predict(pred_dataset=val_df, pred_labels=pred_labels)[TARGET_LABEL]
mu_true = y_true.mean()
mu_pred = y_pred.mean()
balance_of_values = pd.DataFrame({
'Y_true': [mu_true, 1 - mu_true],
'Y_pred': [mu_pred, 1 - mu_pred]
})
print("\nDistribution of Values in Data and Model:")
display(balance_of_values)
performance_summary = pd.DataFrame([single_performance_result])
display(performance_summary)
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/607 [00:00<?, ?it/s]
0%| | 0/607 [00:00<?, ?it/s]
Distribution of Values in Data and Model:
| Y_true | Y_pred | |
|---|---|---|
| 0 | 0.581349 | 0.630952 |
| 1 | 0.418651 | 0.369048 |
| accuracy | precision | recall | f1 | |
|---|---|---|---|---|
| 0 | 0.69246 | 0.716981 | 0.778157 | 0.746318 |
At first glance, this model development process looks promising: with 68.6% accuracy in Combinatorial Purged Cross Validation and 69.8% accuracy on the Validation set, which has 58%/42% split of 1s/0s, the model appears to have a statistical edge. Before we can rely on this edge in live trading, we need to conduct a permutation test to determine if it is statistically significant.
Significance of Accuracy Measurement¶
In [34]:
mu_expected = mu_true * mu_pred + (1 - mu_true) * (1 - mu_pred)
print(f"Expected Mean Accuracy: {mu_expected:.6f}")
accuracy = accuracy_score(y_true=y_true.values, y_pred=y_pred.values)
p, t = permutation_test(
y_pred=y_pred.values,
y_true=y_true.values,
n_iter=int(1e5),
seed=42
)
exp_acc = t.mean()
cts, bins, _ = plt.hist(t, bins=np.arange(0., 1., 0.01))
plt.title(f"Permutation Test of Accuracy\np-value = {p:.2e}")
plt.vlines(exp_acc, ymin=-1, ymax=cts.max(), colors='k', label=f'simulated mean: {exp_acc:.3f}')
plt.vlines(accuracy, ymin=-1, ymax=cts.max(), colors='orange', label=f'model: {accuracy:.3f}')
plt.xlim(0, 1)
plt.ylim(0, None)
plt.ylabel("Frequency")
plt.xlabel("Accuracy")
plt.legend()
plt.show()
percentiles = [0, 1, 5, 10, 20, 50, 80, 90, 95, 99, 100]
accuracy_percentiles = pd.DataFrame({
'percentile': percentiles,
'accuracy': np.percentile(t, q=percentiles)
}).set_index('percentile').T
display(accuracy_percentiles)
Expected Mean Accuracy: 0.521306
| percentile | 0 | 1 | 5 | 10 | 20 | 50 | 80 | 90 | 95 | 99 | 100 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| accuracy | 0.458333 | 0.486111 | 0.496032 | 0.501984 | 0.507937 | 0.521825 | 0.53373 | 0.539683 | 0.545635 | 0.555556 | 0.583333 |
By the accounting for the distribution of values in the data and the distribution in the model's predictions, we determine analytically that our expected accuracy is 52.1%. The permutation test shows that, in 100,000 worlds where market performance is truly random because no inherent structure exists, our model would have achieved accuracies between 45.0% and 58.3% with a median of 52.0% and a mean of 52.1% (as expected). In the real world, we achieved 69.8% accuracy. With a p-value of 0.00001, the permutation test reveals a less than 1 in 100,000 chance that our model achieved this level of accuracy by chance, rather than by representing a legitimate structure in the market.
In order to build a profitable trading strategy, however, we need to balance our win rate (how often our hypotheses are correct) with our reward-to-risk ratio (how big of a dent we make on our P&L when we win compared to when we lose). But before we start simulating a trading strategy, we can learn a lot about reward and risk by looking at the distribution of returns conditioned on our model's predictions.
Visualize Distribution of Predictions and True Returns¶
In [35]:
pred1 = val_df[y_pred==1][TARGET_LABEL]
pred0 = val_df[y_pred==0][TARGET_LABEL]
pred1.hist(bins=np.linspace(-.5, 0.5, 50), label='y_pred=1', alpha=0.5)
pred0.hist(bins=np.linspace(-.5, 0.5, 50), label='y_pred=0', alpha=0.5)
plt.legend()
plt.xlabel(f"True {TARGET_HORIZON_DAYS}-day Return")
plt.ylabel("Frequency")
plt.show()
data = {
'ypred1': [
pred1.mean(),
np.log(pred1 + 1).mean(),
np.log(pred1[pred1 > 0] + 1).sum(),
np.log(pred1[pred1 <= 0] + 1).sum(),
np.log(pred1[pred1 > 0] + 1).sum() + np.log(pred1[pred1 <= 0] + 1).sum()
],
'ypred0': [
pred0.mean(),
np.log(pred0 + 1).mean(),
np.log(pred0[pred0 < 0] + 1).sum(),
np.log(pred0[pred0 >= 0] + 1).sum(),
np.log(pred0[pred0 < 0] + 1).sum() + np.log(pred0[pred0 >= 0] + 1).sum()
]
}
val_returns = pd.DataFrame(
data,
index=[
'mean return',
'mean log return',
'sum log returns: correct',
'sum log returns: wrong',
'net log return'
]
)
val_returns
/Users/ryancardenas/anaconda3/envs/demos/lib/python3.12/site-packages/pandas/core/arraylike.py:399: RuntimeWarning: invalid value encountered in log result = getattr(ufunc, method)(*inputs, **kwargs)
Out[35]:
| ypred1 | ypred0 | |
|---|---|---|
| mean return | 0.021415 | -0.029502 |
| mean log return | 0.024850 | -0.033162 |
| sum log returns: correct | 21.960248 | -16.970933 |
| sum log returns: wrong | -6.205130 | 4.634842 |
| net log return | 15.755117 | -12.336091 |
As hoped, the times when the model predicts positive price changes have a mean return of +2.2% (including the times when the model is wrong); likewise, the times when the model predicts negative price changes have a mean return of -2.9%. In other words, the model is providing a signal that could probably support a profitable trading strategy. To see if that is true in aggregate, we will use logarithms to transform our returns into log returns, which can be added up over time (as opposed to regular returns which must be summed with 1.0 and then multiplied). The "mean log return" accounts for the compounding nature of returns and confirms what the arithmetic mean reveals.
If we consider the sum of log returns whenever the model makes a correct prediction and compare that to the sum of log returns whenever the model's prediction is wrong, we can determine whether the net log return is agreeable. As we can see from our data table, the model has a net log return of +1587% for positive predictions over the four years from Jan 2019 to the end of 2022, while the model has a net log return of -1245% for negative predictions. These net log returns include all the cases where the model predictions were wrong. That's a great sign: we want the true returns to be cumulatively negative whenever the model predicts a "0" (a price decline) and we want the true returns to be cumulatively positive whenever the model predicts a "1".
During computation of the log returns, numpy threw a RuntimeWarning telling us there was an invalid value. This often occurs because of negative values, for which logarithms are undefined. Let's take a look at all the cases in our validation set where the 5-day returns might have been worse that -100%:
In [36]:
val_df[val_df[TARGET_LABEL]<=-1.]
Out[36]:
| target | BZ=F | CL=F | CVX | SHEL | XLE | XOM | ^SPX | A24ATI | CAPG211S | ... | IPG21112S | IPN213111N | INDPRO | OVXCLS | PCU211211 | WTISPLC | PAPR_OPEC | RCRR01NUS_1 | RCRR01R48F_1 | WGFUPUS2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Date | |||||||||||||||||||||
| 2020-04-20 | -2.679161 | 0.009346 | -0.080523 | 0.089761 | 0.088344 | 0.106223 | 0.103959 | 0.026794 | -0.028577 | 0.001298 | ... | -0.000091 | 0.004537 | -0.005934 | -0.166725 | -0.057554 | -0.121349 | 0.066359 | -0.111777 | -0.106001 | 0.045267 |
| 2020-04-27 | -1.339623 | 0.005157 | 0.026667 | 0.002419 | -0.003698 | 0.002316 | 0.006444 | 0.013918 | -0.028577 | 0.001298 | ... | -0.000091 | 0.004537 | -0.005934 | -0.177583 | -0.057554 | -0.121349 | 0.066359 | -0.111777 | -0.106001 | 0.103370 |
2 rows × 22 columns
We can now see exactly where the target variable experienced such a large negative return that log(1 + r) threw a warning: once on the infamous April 20, 2020, when oil prices went negative because COVID caused people to stop driving and oil storage costs were priced higher than the value of the oil itself; and once more on April 27 -- exactly five trading days later -- because even though the absolute return was positive, the percent return was negative due to the negative entry price on April 20. This exceptional market event posed a massive risk to anyone who was fully invested or worse, overleveraged, in oil during that time. The only way I can think of to mitigate that kind of risk in the future is to not allocate too large a fraction of one's portfolio on any single trade or instrument. That said, for the purposes of testing our model in a trading strategy, we will allow full allocation.
Model Performance on Test Set with Toy Trading Strategies¶
Since our primary goal here is to demonstrate the usefulness of HMM-encoded regimes and Bayesian Networks, we won't spend too much time developing a robust trading strategy or strategy development framework. Instead, we will train a model using the same process as before, but this time on the combined Train-Val sets. We will then retrieve the model's predictions over the entire Test set and simulate the realized profit-and-loss for two strategies: a simple long-only strategy and a long-short strategy.
The rules for the long-only strategy are as follows:
- If model predicts a price increase, go long with a 20% allocation of portfolio funds (since the prediction horizon is 5 days, so we can have up to 5 concurrently open trades). Exit positions at the close of the fifth day.
- If the model predicts a price decrease, do nothing. Do not close open positions; do not open new positions.
The rules for the long-short strategy are as follows:
- If model predicts a price increase, go long with a 20% allocation of portfolio funds (since the prediction horizon is 5 days, so we can have up to 5 concurrently open trades). Exit positions at the close of the fifth day.
- If the model predicts a price decrease, go short with a 20% allocation of portfolio funds. Exit positions at the close of the fifth day.
We will model trading fees, slippage, and margin interest, as a bundled per-trade expense. As of September 1st, 2025, Schwab's pricing table shows a 2.25 USD per-contract fee for futures while Yahoo Finance lists CL=F at 64.92 USD per barrel for contracts of 1000 barrels each, so the 4.50 USD round-trip trading fee on a single 64,920 USD contract represents 0.007%. Combine that with Schwab's ~12% annualized margin interest rate over the 5 days each position will remain open, we arrive at fees representing 0.17% of each trade (Schwab's 2.25 USD trading fee is neglible). Tack on +/- 5 cents of average slippage on each trade for a total 0.25%. We will round this up to 0.3% per trade to be on the safe side.
$$\text{Trading Fees} + \text{ Margin Interest} + \text{ Slippage} = \frac{4.50}{64920} + 5 * \frac{0.12}{360} + \frac{0.05}{64.92} = 0.0025 \approx 0.3\% \text{ per trade}$$
For simplicity, we will allow fractional positions in all strategies. We will not model stop orders, unrealized P&L due to intraday price action, or margin maintenance requirements. Concurrent long and short positions are allowed, because traders or firms may manage long/short positions on the same instrument in seperate accounts.
Retrain on Train + Val¶
In [37]:
####################################################################################
# Define Settings
####################################################################################
# Settings for encoder
features_to_encode = yf_features
# Settings for HMM: hidden state transitions, emissions, and starts
hmm_verbosity = 0
edges = [
[0.90, 0.05, 0.05],
[0.05, 0.90, 0.05],
[0.05, 0.05, 0.90]
]
hs_bear = Categorical([[0.6, 0.4]])
hs_stag = Categorical([[0.5, 0.5]])
hs_bull = Categorical([[0.4, 0.6]])
hidden_states = [hs_bear, hs_stag, hs_bull]
starts = [0.3, 0.3, 0.4]
# Settings for Hill Climb Search
max_indegree = 4
expert_edges = [
('^SPX', 'XLE'), # Market influences the sector in a top-down fashion
('PCU211211', 'CPIENGSL'), # Producer price probably influences consumer prices
('IPN213111N', 'CAPG211S'), # Drilling influences capacity
]
# Settings for prediction
pred_labels = [TARGET_LABEL]
# Settings for returned values
return_model = True
####################################################################################
# Iterate Through Combinatorial Cross Validation (CPCV) Train and Val Sets
####################################################################################
train_val_data = pd.concat([train_df, val_df]).iloc[:-TARGET_HORIZON_DAYS]
mdp = ModelDevelopmentPipeline(
features_to_encode=features_to_encode,
hmm_verbosity=hmm_verbosity,
edges=edges,
hidden_states=hidden_states,
starts=starts,
max_indegree=max_indegree,
expert_edges=expert_edges,
pred_labels=pred_labels,
return_model=return_model
)
mdp.fit(train_dataset=train_val_data)
single_performance_result = mdp.evaluate(pred_dataset=test_df, pred_labels=pred_labels)
y_true = mdp.discretizer.transform(test_df[pred_labels])[TARGET_LABEL]
y_pred = mdp.predict(pred_dataset=test_df, pred_labels=pred_labels)[TARGET_LABEL]
mu_true = y_true.mean()
mu_pred = y_pred.mean()
balance_of_values = pd.DataFrame({
'Y_true': [mu_true, 1 - mu_true],
'Y_pred': [mu_pred, 1 - mu_pred]
})
print("\nDistribution of Values in Data and Model:")
display(balance_of_values)
performance_summary = pd.DataFrame([single_performance_result])
display(performance_summary)
0%| | 0/1000000 [00:00<?, ?it/s]
0%| | 0/437 [00:00<?, ?it/s]
0%| | 0/437 [00:00<?, ?it/s]
Distribution of Values in Data and Model:
| Y_true | Y_pred | |
|---|---|---|
| 0 | 0.508955 | 0.437313 |
| 1 | 0.491045 | 0.562687 |
| accuracy | precision | recall | f1 | |
|---|---|---|---|---|
| 0 | 0.686567 | 0.723549 | 0.621701 | 0.66877 |
Significance of Accuracy on Test¶
In [38]:
mu_expected = mu_true * mu_pred + (1 - mu_true) * (1 - mu_pred)
print(f"Expected Mean Accuracy: {mu_expected:.6f}")
accuracy = accuracy_score(y_true=y_true.values, y_pred=y_pred.values)
p, t = permutation_test(
y_pred=y_pred.values,
y_true=y_true.values,
n_iter=int(1e5),
seed=42
)
exp_acc = t.mean()
cts, bins, _ = plt.hist(t, bins=np.arange(0., 1., 0.01))
plt.title(f"Permutation Test of Accuracy\np-value = {p:.2e}")
plt.vlines(exp_acc, ymin=-1, ymax=cts.max(), colors='k', label=f'simulated mean: {exp_acc:.3f}')
plt.vlines(accuracy, ymin=-1, ymax=cts.max(), colors='orange', label=f'model: {accuracy:.3f}')
plt.xlim(0, 1)
plt.ylim(0, None)
plt.ylabel("Frequency")
plt.xlabel("Accuracy")
plt.legend()
plt.show()
percentiles = [0, 1, 5, 10, 20, 50, 80, 90, 95, 99, 100]
accuracy_percentiles = pd.DataFrame({
'percentile': percentiles,
'accuracy': np.percentile(t, q=percentiles)
}).set_index('percentile').T
display(accuracy_percentiles)
Expected Mean Accuracy: 0.498877
| percentile | 0 | 1 | 5 | 10 | 20 | 50 | 80 | 90 | 95 | 99 | 100 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| accuracy | 0.41194 | 0.453731 | 0.465672 | 0.474627 | 0.483582 | 0.498507 | 0.516418 | 0.522388 | 0.531343 | 0.543284 | 0.58209 |
As with the Validation set, the model's accuracy metric on the Test set is statistically significant to the 0.00001 level, according to permutation testing.
Visualize Distribution of Predictions and True Returns¶
In [39]:
pred1 = test_df[y_pred==1][TARGET_LABEL]
pred0 = test_df[y_pred==0][TARGET_LABEL]
pred1.hist(bins=np.linspace(-.5, 0.5, 50), label='y_pred=1', alpha=0.5)
pred0.hist(bins=np.linspace(-.5, 0.5, 50), label='y_pred=0', alpha=0.5)
plt.legend()
plt.xlabel(f"True {TARGET_HORIZON_DAYS}-day Return")
plt.ylabel("Frequency")
plt.show()
data = {
'ypred1': [
pred1.mean(),
np.log(pred1 + 1).mean(),
np.log(pred1[pred1 > 0] + 1).sum(),
np.log(pred1[pred1 <= 0] + 1).sum(),
np.log(pred1[pred1 > 0] + 1).sum() + np.log(pred1[pred1 <= 0] + 1).sum()
],
'ypred0': [
pred0.mean(),
np.log(pred0 + 1).mean(),
np.log(pred0[pred0 < 0] + 1).sum(),
np.log(pred0[pred0 >= 0] + 1).sum(),
np.log(pred0[pred0 < 0] + 1).sum() + np.log(pred0[pred0 >= 0] + 1).sum()
]
}
test_returns = pd.DataFrame(
data,
index=[
'mean return',
'mean log return',
'sum log returns: correct',
'sum log returns: wrong',
'net log return'
]
)
test_returns
Out[39]:
| ypred1 | ypred0 | |
|---|---|---|
| mean return | 0.020401 | -0.016918 |
| mean log return | 0.019299 | -0.017883 |
| sum log returns: correct | 8.293359 | -9.767372 |
| sum log returns: wrong | -2.638829 | 3.025452 |
| net log return | 5.654530 | -6.741920 |
As with the Validation set, the model's predictions tend to match the skew of the market's returns.
Simulate Long-Only and Long-Short Strategies Guided by Model's Predictions¶
In [40]:
# Download FRED's 4-week treasury bill secondary market rate as risk-free rate
rf_percent = FRED_SESSION.get_series('DTB4WK').sort_index(ascending=True).ffill()
In [41]:
wti_price = yf_df['CL=F'].loc[y_pred.index]
rf_annualized = rf_percent.loc[wti_price.index].values / 100.
n_tests = 3
fees_per_trade = 0.003
returns_long_only = np.full(shape=y_pred.shape, fill_value=np.nan)
returns_long_short = np.full(shape=y_pred.shape, fill_value=np.nan)
for i in range(y_pred.shape[0]):
if y_pred.iloc[i] == 1:
returns_long_only[i] = test_df.loc[y_pred.index[i], TARGET_LABEL] - fees_per_trade
returns_long_short[i] = test_df.loc[y_pred.index[i], TARGET_LABEL] - fees_per_trade
elif y_pred.iloc[i] == 0:
returns_long_only[i] = 0. # No trade, no fees!
returns_long_short[i] = -test_df.loc[y_pred.index[i], TARGET_LABEL] - fees_per_trade
else:
raise ValueError(f"Predictions should be either 1 or 0; received {y_pred.iloc[i]}")
# The buy-and-hold baseline strategy follows the cumulative percent returns of CL=F.
pnl_buy_and_hold = wti_price.ffill() / wti_price.iloc[0]
metrics_buy_and_hold = get_portfolio_metrics(
portfolio=pnl_buy_and_hold.values,
benchmark_portfolio=pnl_buy_and_hold.values,
n_tests=n_tests,
rf_annualized=rf_annualized,
bars_per_year=BARS_PER_YEAR
)
# Each return is multiplied by (1 / TARGET_HORIZON_DAYS) to represent the fact that we
# are allocating the same fraction of available funds to each open trade.
pnl_long_only = pd.Series(
data=((returns_long_only / TARGET_HORIZON_DAYS) + 1).cumprod(),
index=y_pred.index
)
metrics_long_only = get_portfolio_metrics(
portfolio=pnl_long_only.values,
benchmark_portfolio=pnl_buy_and_hold.values,
n_tests=n_tests,
rf_annualized=rf_annualized,
bars_per_year=BARS_PER_YEAR
)
# Each return is multiplied by (1 / TARGET_HORIZON_DAYS) to represent the fact that we
# are allocating the same fraction of available funds to each open trade.
pnl_long_short = pd.Series(
data=((returns_long_short / TARGET_HORIZON_DAYS) + 1).cumprod(),
index=y_pred.index
)
metrics_long_short = get_portfolio_metrics(
portfolio=pnl_long_short.values,
benchmark_portfolio=pnl_buy_and_hold.values,
n_tests=n_tests,
rf_annualized=rf_annualized,
bars_per_year=BARS_PER_YEAR
)
plt.figure(figsize=(12, 6))
plt.plot(pnl_buy_and_hold, label=f"CL=F Buy-and-Hold: {100 * metrics_buy_and_hold['final_pnl']:.1f}%")
plt.plot(pnl_long_only, label=f"Long-Only P&L: +{100 * metrics_long_only['final_pnl']:.1f}%")
plt.plot(pnl_long_short, label=f"Long-Short P&L: +{100 * metrics_long_short['final_pnl']:.1f}%")
plt.title("Cumulative Returns")
plt.ylabel("Portfolio Growth Factor")
plt.xlabel("Date")
plt.grid()
plt.legend()
plt.show()
metrics_df = pd.DataFrame(
[metrics_buy_and_hold, metrics_long_only, metrics_long_short],
index=['buy_and_hold', 'long_only', 'long_short']
)
display(metrics_df)
| final_pnl | max_drawdown | calmar | omega | sharpe | psr | dsr | |
|---|---|---|---|---|---|---|---|
| buy_and_hold | -0.167945 | -0.390158 | -0.293722 | 0.967604 | -0.203634 | 0.5 | 0.005528 |
| long_only | 1.757774 | -0.118229 | 3.504147 | 2.404964 | 3.607103 | 1.0 | 1.000000 |
| long_short | 6.772678 | -0.118229 | 9.370632 | 2.504880 | 5.622200 | 1.0 | 1.000000 |
Between Jan 2023 and Sept 2025, the buy-and-hold strategy on WTI Oil Futures (where we roll the position at the end of each month when the futures expire) would have lost 16.8%, but the model-informed long-only and long-short strategies would have gained 160% and 598%, respectively. Not only did our model produce higher returns, it also reduced the max drawdown from 39.0% down to 11.8% -- less than a third! With comparatively high Calmar and Omega ratios and large, positive Sharpe ratios (in contrast to the baseline's negative Sharpe), both model-informed strategies far exceeded the market's performance. The high Probabilistic and Deflated Sharpe Ratios confirm that, to an arbitrary level of confidence, we can say that our true Sharpe ratios beat the baseline and the noise.
Project Summary¶
We explored how to use Hidden Markov Model's to encode stock market regimes and how to use Bayesian Networks to predict price movements for oil futures five days in advance. In the process, we:
- Took macro-economic and stock market data from the Yahoo Finance, FRED, and EIA databases, computed percent changes for each series, and shifted each series forward to appropriately capture the lag between the period each data point represents and the time when data becomes available for use in trading.
- Split the data into Train, Validation, and Test sets, performed Combinatorial Purged Cross Validation within the Train set to identify a promising model development process, evaluated that process on the out-of-sample Validation set, and concluded that our process was worth testing in a paper trade simulation.
- Re-ran the model development process from scratch on the combined Train + Validation sets, and then evaluated the model's performance on the out-of-sample Test set by computing classification metrics (accuracy, precision, recall, f1-score), examining the conditional distributions of true returns given the model's predictions, conducting a permutation test, and comparing the profit-and-losses of two different trading strategies against a baseline buy-and-hold strategy.
Our model provided a statistically significant trading signal that gave two basic strategies enough "edge" to drastically outperform the market from 2023 to late 2025, deploying one-fifth of the portfolio per trade (per day). In the end, these results demonstrated the potential for HMM-informed Bayesian Networks to identify true patterns in oil markets.
In [ ]: